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ABSTRACT 

Over m,000 high school seniors were studied with 
respect to sociocultural differences on cognitive test item 
responses. Six different cognitive tests and ten different groups 
were analyzed. The tests were: vocabulary, picture-number, reading, 
letter-groups^ mathematics, and mosaic comparisons. The groups were: 
American Indians, blacks, Mexican-Americans, Puerto Ricans, other 
Latin-Americans, Oriental-Americans^ vhite IJortheastern, white North 
Central, white Southern, and white Western. Proportions of each group 
responding correctly to each item of each test werv-:: computed and then 
transformed to equal interval scales of delta-values. The 
delta-values for the white North Central group were then 
cross-plotted with each of the other groups to yield an elliptical 
pattern of points for each comparison. The major axis of the ellipse 
for each cross-plot was determined and the distance of each item 
point from it computed. These distances were used to create a vector 
index of cross-cultural stability. Items having notable patterns of 
instability were examined closely for factors which might explain the 
instabilities. It was concluded that certain vocabulary iteais were 
unstable for some Spanish-speaking groups. These vocabulary 
instabilities were attributed to coTnci'iaental cognate influences 
operating to make some English vocabulary items relatively easier for 
the Spanish-speaking groups. It was also observed that reading test 
items containing material relevant to black culture were relatively 
easier for blacks than were other items in the test battery.. 
(Author/RC) 
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Abstract 



A national random sample of over 14,000 high school seniors was 
studied with respect to socio-cultural differences on cognitive test 
item reponses. Six different cognitive tests and ten different groups 
were analyzed. The tests were: vocabulary, picture-number, reading, 
letter-groups, mathematics, and mosaic comparisons. The groups were: 
American Indians, blacks, Mexican-Americans, Puerto Ricans, Other Latin- 
Americans, Oriental-Americans, white Northeastern, white North Central, 
white Southern, and white Western, Proportions of each group responding 
correctly to each item of each test were computed and then transformed 
to equal interval scales of delta-values. The delta-values for the white 
North Central group were then cross-plotted with each of the other groups 
to yield an elliptical pattern of points for each comparison. The major 
axis of the ellipse for each cross-plot was determined and the distance 
of each item point from it computed. These distances were used to c;r:(ea^e 
a vector index of cross-cultural stability. Items having notable patterns 
of instability were examined closely for factors which might explain the 
instabilities. It was concluded that certain vocabulary items were un- 
stable for some Spanish-speaking groups. These vocabulary instabilities 
were attributed to coincidental cognate influences operating to mke some 
English vocabulary items relatively easier for the Spanish-speaking groups. 
It was also observed that reading test items containing material relevant 
to black culture were relatively easier for blacks than were other items 
in the test battery. Further interpretive analysis was considered to be 
beyond the scope of the study. 
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The research reported in this paper was supported in its entirety 
by a Small Grant from the National Institute of Education's Field Initi- 
ated Studies- Program. Because of the limited funding associated with 

these Small Grants the studies themselves must be of a very limited 
nature. The present study is only an exploration into the many-faceted 
problem of testing in a society become increasingly concerned about 
equality. -Since these concerns have been at the forefront of our nation's 
focus for several years now, a number of other studies have treated the 
associated testing problem in some depth and at a much greater expense 
than was possible here. Accordingly, it is hoped that invidious compari- 
sons will not be made with other, more elaborate, efforts. Rather than 
being a comprehensive analysis, the study is intended merely to suggest 
a new way of thinking about test use and construction. The data used 
have the potential for much more exhaustive analysis, but this was not 
possible in view of the limted funding. Hopefully, the analyses reported 
will represent only a small beginning toward an understanding of this rich 
set of data. 
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There is * a growing conviction among members of ethnic minority groups 

and others that traditional tests of academic achievement and tests used ^ 

for employment decisions are biased in favor of a white middle-class culture. 

Although much research tends to discount such a belief (e.g., Stanley, 1971; 

Cleary, 1968; Rock, 1970; Campbell, Crooks, Mahoney, and Rock, 1973), there 

is no doubt that some items on some tests are more difficult for some socio- 

cultural groups. This point was emphasized by Green and Draper (1972): 

As a matter of fact we do know that most academic tests, both 
aptitude and achievement, yield consistently higher scores for one, 
set of groups in society in contrast to various other groups such 
as poor people, blacks, and Chicanos (Coleman, 1966). Some people 
overgeneralize these results to indicate that the latter groups are 
inferior to the former. In so doing they are assuming the tests are 
fair and unbiasad. [p. 5] 

What the words "fair" and "unbiased" mean, however, has proven to be diffi- 
cult to define. Green and Draper note: 

A biased test is popularly understood to be a test which is 
unfair to identifiable subgroups of the general population in which 
it is being used. Although many people seem to believe the matter 
is simple, little is actually known about the nature of bias in tests 
and even the most widely accepted propositions badly need verification. 

[p. 1] 

This kind of confusion led Darlington (1971) to propose that the concept of 

"cultural fairness" be replaced by a concept of "cultural optimality." No 

terminology, however, replaces the need for a careful consideration of the 

way in which a specific test (or portion thereof) is used. In the words of 

Thorndike (1971) : 

Since there are many different uses that can be made of a 
particular test or inferences that can be based upon it, it is 
entirely possible that one use or inference is fair while another 
is grossly unfair. [p. 63] 

As an example of unfair test use, Thorndike presented the following hypothetical 
Item: 
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The usual temperature for baking a cake is about: 
(A) 250° (B) 300° (C) 350' (D) 400° 
In terms of the proportion of correct responses that would probably be obtained 
were such an item administered, the item would seem unfair to males, since they 
spend less time cooking than do females. Thus, if this item were used as part 
of a college admissions test, it would clearly seem inappropriate. But if it 
were used as part of a test to select persons for employment as bakers, then 
the item would at least have face validity. 

Rather than considering individual items, the more common approach has 
been to consider entire tests. Attempted statistical definitions of test bias 
have been approached by two basically different methods: those using criteria 
external to the test, and those using only internal criteria (Potthoff, 1972). 
Methods employing external criteria generally involve use of test scores for 
prediction of some future success. Cleary (1968) has provided a widely ac- 
cepted definition of test bias which compares regression equations of test , 
scores on criterion for different groups. She states: 

A test is biased for members of a subgroup of the population 
if, in the prediction of a criterion for which test was designed, 
consistent nonzero errors of prediction are made for members of 
the subgroup. In other words, the test is biased if the criterion 
score predicted from the common regression line is consistently^ 
too high or too low for members of the subgroup. With^^this defi- 
nition of bias, there may be a connotation of "unfair ," ^particu- 
larly if the use of the test produces a prediction that is too 
low. 

Thorndike (1971) demonstrated that a test which would be fair by Cleary 's 
definition may be unfair by another standard. He has shown that when mean 
differences between two groups on the predictor are large relative to mean- 
differences on the criterion, even when regression lines for the two groups 
are equal, the test would select a smaller proportion of the low scoring 
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group than th,e proportion who actually could have been successful on the 
criterion. According to Thorndike (1971) a fair test must select a pro- 
portion of the minority group which is equal to the proportion who would 
actually succeed on the criterion. This may necessitate the lowering of 
critical cut-off points for selection for some minority groups. 

Cole (1972) approaches test bias by examining decision errors for 
various groups. The proportion of false positives (those with acceptable 
predictor scores and unacceptable criterion scores) to false negatives 
(those with unacceptable predictor scores and successful criterion scores) 
should be the same in all groups if the test is "fair." 

Comparative studies of these and other definitions of test bias (Linn, 
1973; Darlington, 1971) show that the previously mentioned approaches are 
contradictory. The enigmas encountered in the external criterion approach 
have been further clarified by Reilly (1973) . A test which may be fair 
by one definition may be unfair by another. Therefore, a single statisti- 
cal solution to the problem of test bias derived from the comparison of 
tests to an external criterion seems impossible. Considering that the 
external criterion, itself, may be biased (see Campbell, et al.,1973) 
leads one to question this entire approach. Both Darlington (1971) and 
Linn (1973) conclude that statistical solutions alone are not sufficient 
to solve the test bias problem — some value judgments must be made. 
Darlington proposes the concept of the "culturally optimal test" which 
balances cultural differentiation with validity and which employs both 
subjective policy level decisions and empirical statistical information. 

An alternate approach to the definition of cultural bias attempts 
to make some statistical statement about the items in a test without 
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information other than that obtainable from the test items themselves. • The 
methods used by Cleary and Hilton (1968), Echternacht (1972a, 1972b), Angoff 
and Ford (1973) , Angoff (1972) , Angoff and Modu (1973) , and Cardall and Cof fman 
(1964) essentially compare item difficulties across cultural groups. Those 
items which are either unusually easy or unusually difficult for one group in 
relation to another group are examined. If many items are unusual across 
several group comparisons, the test is said to be biased. \^ile these studies 
are labeled studies of "item bias," they rarely attempt to analyze sources of 
deviation for outstanding items. The attempt has been -usually to make some 
inference about the test as a whole by demonstrating the existence or lack of 
existence of a significant item by group interaction. Individual items are 
not considered in a subjective sense nor are the possible sources of bias in 
any one individual item explored. 

Those few papers that do consider individual items most often attempt to 
define some mechanical procedure with which "biased items" may be detected. 
Given the importance of the use to which items (or entire tests) are put, no 
entirely mechanical procedure would seem likely to gain acceptance. It is 
the objective of the present study to explore the problem of cross-cultural 
stability of test items with a combined mechanical and subjective approach 
in much the same way as Darlington combined the mechanical and subjective 
in thinking about entire tests. The procedure is, first, to compute an index 
(mechanically) which is useful in- detecting especially unstable items and 
then to apply subjective analyses to determine what, if anything, character- 
izes these items. Similarly, items may also be detected because of their 
apparent stability. Subjective judgments are then used to characterize these 
items * 
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The procedure used is similar to that used by Angof f and Ford (1971) . 
They compared several samples of black and white students drawn from the 
1970 PSAT (Preliminary Scholastic Aptitude Test) administration in Georgia. 
Item analyses were conducted for each sample and item difficulty cross- 
plotted for pairs of samples. As a measure of item x group interaction, 
they used the correlation of item difficulties (the lower the correlation, 
the more the interaction) . Angof f and Ford concluded that the findings 
were sufficiently provocative to deserve more detailed study. Like the 
cross-plots of Cleary and Hilton, .those of Angof f and Ford also indicated 
a number of items that appeared to be especially difficult for blacks. 
Angoff and Ford suggested a need for studies with larger samples than those 
they had used. This remark of theirs is of special interest: 

Further editorial examination of the items that were especially 
harder for the blacks suggested, as one would expect, particular 
difficulties with vocabulary and concepts pertaining to unfamiliar 
places and experiences, and possibly also to confusion with special 
meanings and significances characteristic of the ghetto. 

^ The same kinds of phenomena were discussed by Taylor (1971) in an 
entirely different context (that of speech difficulties) and from a different 
disciplinary viewpoint (socio-linguxstics) . Deemphasizing the importance of 
the ghetto, Taylor traced the evolutionary history of Black English and showed 
how, because of a different long-term cultural development. Black English is 
very different from what is sometimes called Standard English. The position 
of Taylor is important in that it challenges the so-called social deprivation 
theory (that blacks simply have underdeveloped language and cognitive abil- 
ities) by emphasizing that Black English (implying a language more deeply 
rooted than "hip talk") is very different from Standard English. Taylor 
also Indicated that there are probably several types of English among whites 
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living in the continental United States. Southern White English has strong 
similarities to Black English; however, Standard English is very different 
from either Black English or Southern ^fliite English. Since it is well known 
that white southerners also tend to score low on standard achievement tests, 
this observation of Taylor is especially noteworthy. 

If strong differences exist between Black English and Standard English, 
and even among different types of American English, then the linguistic pat- 
terns of American Indians, Mexican-Americans, Puerto Ricans, and Oriental- 
Americans might be expected to differ as well. Armstrong (1972) had members 
of several ethnic groups rate test items as to the degree they believed them 
to be biased against their group. Within ethnic groups, he found surprising 
agreement on which items were biased. But the items considered to be biased 
varied considerably from one ethnic group to another. Accordingly, Armstrong's 
research would support Taylor's theory. Armstrong, however, conducted no 
analyses of data from test administrations for these different ethnic groups. 

A recently collected, and extensive, set of data has afforded the oppor- 
tunity for conducting the present study. These data are those from the 
National Longitudinal Study of the High School Class of 1972."^ In this 
sample of over 17,000 high school students, special attention was paid to 
the problem of insufficient numbers of cases for minorities so that adequate 
data were obtained for analyses relating to minorities. With such a large 
number of cases and detailed classifications by ethnic group, region, and 
other identifying criteria, these data are ideal for the study of the problem 
of cross-cultural stability of test items. The cognitive tests used in the 



^Conducted for the U.S. Office of Education by the Educational Testing Service 
under Contract No. EC-0-72-0903 
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National Longitudinal Study (NLS) cover a wide range of abilities and item 
presentation styles. Moreover, the care with which the sample was taken 
offers the potentiality of generalization to the nation as a whole. 

An exploratory development of a procedure for computing an index of 
cross-cultural stability was conducted. Some characteristics of items so 
detected, obtained from a subjective analyses of the items, are presented. 
The possibility of using the same technique, cumulatively, to describe 
entire tests with respect to their cross-cultural stability is also consid- 
ered. Beyond a brief consideration of the causal factors underlying the 
item instabilities presented, a need exists for generalizations about these 
causal factors. It is believed that these generalizations are best attempted 
by those who belong to the specific socio-cultural group to which the insta- 
bilities relate or by those, such as socio-linguists , who have studied such 
problems. Accordingly, it is hoped that the ethnic scholars and others to 
whom this report is being disseminated will attempt these generalizations. 
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The Sample 

The data used in the project were recently collected as a part of the 
NLS, This study was based on a stratified two-stage probability sample. 
Schools were selected nationwide, with known probabilities, by WESTAT. Cor- 
poration, from universe listings of schools retained by the U, S, Office 
of Education. The population was stratified by a set of eight variables: 
(1) public or nonpublic, (2) geographic region, (3) enrollment size class, 
(4) proximity to institutions of higher education, (5) percent minority, 
(6) income, level of the community around the school, (7) school type — 
where Type A represented schools of low income or high minority classi- 
fication, and Type B represented all others — and (8) degree of urbanization. 
Altogether, 600 final strata were defined and Type A schools were selected 
at twice the sampling rata of Type B schools to produce a final sample of 
1,200 schools, two from each final stratum. ' 

Within each cooperating sample school, a random sample of students in 
grade 12 (or its equivalent) was taken by Educational Testing Service from 
lists of all such students provided by the school. Where possible, 18 
students were selected. Occasionally, noncooperating students or small 
school enrollments resulted in fewer than 18 students being included in 
the final sample. 

A few kinds of schools and students were excluded from the study. 
Excluded schools consisted mostly of schools for physically or mentally 
handicapped students, schools for legally confined students, and schools 
which did not enroll students of their own (such as area vocational schools 
having students enrolled in other schools) . Included schools were required 
to be within the 50 states and the District of Columbia. Excluded students 
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consisted of early graduates, adult education students, and students who in 
the view of their school would be harmed by the experience of the project. 
The final count of students involved in the study was 17,726, and these 
represented 1,044 different high schools. 
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Inst rumen ts 

The instruments examined were those used in the National Longitudinal 
Study Sample. Test items from the battery are included as Appendix A of 
this report. Appendices B and C, respectively, are the Answer Sheet used 
with the test battery and the Survey Administrators Guide (which describes 
the procedures used in administering the test) . The sequential order of 
the tests, described in detail below, was: Vocabulary, Picture-Number, 
Reading, Letter Groups, Mathematics, and Mosaic Comparisons. This sequence 
was chosen because it interspersed the three more conventional and the 
three more novel tests, an arrangement that provides interest and motiva- 
tion for the examinees. Vocabulary was chosen for the first position be- 
cause of the inherent simplicity of this test's format and directions. At 
the outset, it was believed that the Vocabulary Test should build the con- 
fidence of the students in their capability to perform well. Because it 
iG quite speeded. Mosaic Comparisons was placed last to prevent any anxiety 
that might be engendered by this speededness from persisting in later test 
sections. 

The composition of the NLS battery represented a balancing of somewhat 
opposing considerations. The primary objective was to obtain a comprehensive 
description of persons whose backgrounds, ethnic affiliations , and socio- 
economic status are quite diverse. At the same time, the need for various 
measures had to be balanced with the requirement of using a battery of 
reasonable length. Lengthy tests are a nuisance to schools that must 
schedule time to administer them and to students who must endure them 
without significant fatigue or loss of interest. For this reason,- the 
battery was held to 69 minutes of testing time plus 36 minutes of 
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administrative time for a total of 105 minutes. Table 1 provides a summary 
of the properties of the battery. 

An ETS optical scanning system (SCRIBE) was used in processing the 
test answers, which students entered on a separate answer sheet (Appendix 
B). This process thus maintained a uniform procedure with previous use of 
the test components, rather than introducing another process variable by 
having students indicate answers by some other procedure, such as circling 
or directly marking answers in test books. 

To conserve testing time, some of the tests used in the NLS battery 
were shorter than the parent versions from which they were derived. The 
tests were originally designed to yield reliability coefficients appro- 
priate for use with individual students. However, the tests used in NLS 
were not intended for making decisions about individuals; rather, they 
were used as group measures, wherein the reliability of the mean scores 
for various samples or subgroups was the critical consideration. In such 
situations, the error variance of a mean is only 1/N^^ the error variance 
associated with an individual score. Thus, the tests in the battery, with 
estimated reliabilities ranging from .70 to .85, yield highly reliable 
measurements of the mean. A brief description of each test section and 
a summary of its psychometric properties follow. 

Vocabulary . A brief test using synonym format consisting of items 
drawn from the longer Project Access Vocabulary Test. The 15 Items selected 
were intended to avoid academic or collegiate bias and to be of an appro- 
priate level of difficulty for the NLS twelfth grade population. Verbal 
ability is known to be related to performance in most academic pursuits 
as well as professional and semiprof essional occupations. The straight- 
forward vocabulary synonym test is the best and most well-documented 
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measure of this verbal ability. Evidence for the predictive validity of 
the Vocabulary Test is given in Table 2. Median correlations between 
Vocabulary Test scores and first-term grade-point averages of students 
enrolled in various curriculums within two-year community colleges are 
in the range of 20 to 40. The Vocabulary Test has also been related to 
subsequent performance in specific entry-level English courses at com- 
munity colleges (Ford, 1970). The median correlation represented between 
the Vocabulary Test and freshman English courses within 38 community col- 
leges was .26 with a range of validity of .08 to .40, depending upon the 
specific college considered. 

Picture-Number . Consists of a series of drawings of familiar objects, 
each paired with a number. The student, after studying the picture-number 
pairs, is asked to recall the number associated with each object. This 
test appeared in both the CGP and Project Access batteries. The inclusion 
of the Picture-Number Test represents acknowledgment of a line of research 
that suggests that populations low in economic status have relatively higher 
mean scores in associative memory than in other types of ability (Semler & 
Iscoe, 1963; Rohwer, et al., 1968; Jensen, 1969). Further recent theoreti- 
cal developments would suggest that such abilities can be utilized in in- 
creasing the school achievement of this same group (Rohwer, 1971). Pre- 
dictive validity information is not currently available. However, the test 
does have face validity based on the references cited above. 

Reading . Based on short passages (100-200 words) with several related 
questions concerning a variety of reading skills (analysis, interpretation) 
but focusing on straightforward comprehension. The Reading Test draws upon 
items of particular relevance to minority group students taken from the 
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Project Access Reading Test. The Reading Test was included for two purposes. 
It is a direct measure of skill used widely throughout the educational system 
and thus has good face validity and widely-recognized importance. The pattern 
of validity coefficients for the Reading Test against the criterion of fresh- 
man grade average is, as would be expected, similar to that for vocabulary 
(Table 2). 

Letter Groups . This test requires the student to draw general concepts 
from sets of data or to form and try out hypotheses in a nonverbal context. 
The items consist of five groups of letters among which four groups share a 
common characteristic while the fifth group is different. The student indi- 
cates which group differs from the others. As a test of inductive reasoning, 
the test measures one of the four aptitudes (verbal, quantitative, reasoning, 
and spatial/perceptual) which have considerable precedent in representing 
the varieties of cognitive skills. Tests of inductive reasoning have, in 
particular, been shown to be useful in research involving minority ethnic 
groups (Lesser, Fifer, & Clark, 1965; Stodolsky & Lesser, 1967; Flaugher, 
1971). This test, in combination with the Mathematics Test that was included 
in the battery, provided a measure of the reasoning capacity of students. 
Unlike the Mathematics Test, however, the Letter Groups Test seems to be less 
dependent on knowledge obtained in a formal education setting; thus, it may 
offer an opportunity for inner-city and minority group students who come from 
poor educational backgrounds to demonstrate their reasoning abilities without 
regard for the lack of educational and mathematical training they may have 
had. The predictive validity for the Letter Groups Test is particularly 
good for predicting performance in courses involving business and office- 
related occupational training. A median correlation of .37 among 11 
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coimnunity colleges was found between the Letter Groups Test and freshman 
grade performance in occupational-technical programs in business (Table 2) , 

Mathematics . Consists of quantitative comparisons in which the student 
indicates whether two quantities are unequal (and which is greater) , unequal 
or not ascertainable from the information given. This type of item is 
relatively quickly answered and provides measurement of basic competence 
in mathematics while minimizing the amount of time required for actual 
computation. The test is a shortened version of the Project Access instru- 
ments but omits those items that tap algebraic, geometric, or trigonometric 
skills'. The parent test from which the NLS Mathematics Test was derived 
shows potency in predicting community college course grades (Ford, 1970) • 
In addition, the predictive validity of the SAT and Mathematics Test, 
mathematics scores in the NLS battery may be linked, has been thoroughly 
documented (Angoff, 1971). 

Mosaic Comparisons . Measures perceptual speed and accuracy through 
items which require that small differences be detected between pairs of 
otherwise identical mosaics or tile-like patterns. A deliberately speeded 
test, it has three separately timed sections consisting of increasingly more 
complex mosaic patterns. 'Mosaic Comparisons represents another of the funda- 
mental measures used in many studies of aptitudes among minority groups. 
Tests like this which represent the spatial/perceptual domain seem, more 
than tests in the other domains considered, to allow students from minority 
groups an opportunity to perform better than majority group students. The 
Mosaic Comparisons Test, unlike many other spatial/perceptual scanning 
measures, is simple for the student to understand. Its predictive validities 
with performance in occupational-technical two-year career and one-year 
career business programs at the community college ^level were .28 and .42 
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respectively. These correlations represent median Values across a number 
of different colleges. In addition to predictive validity in the career 
business '^rea, the Mosaic Comparisons Test has shown a median correlation 
of .23 with the freshman grade performance of students enrolled in college- 
parallel fine arts curriculum. 
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Method 

The sample was first divided into ten mutually exclusive groupings 
as follows: 

1. American Indian 

2. Blac^ or Afro-American or Negro 

3. Mexican- American or Chicano 

4. Puerto Rican 

5. Other Latin-American or of Spanish Origin 

6. Oriental or Asian-American 

7. Northeastern White or Caucasian (10% random sample) 

8. Southern White or Caucasian (10% random sample) 

9. North Central White or Caucasian (10% random sample) 
10. Western White or Caucasian 

Item analyses were then conducted for each group of subjects so defined. 
The item analyses were performed using a procedure outlined by Angoff & 
Ford (1973). In this procedure, the proportion of a sample or sub-sample 
' answering a given item correctly (known as the **2-value") is first calcu- 
lated. The p-values are then transformed to an equal interval scale by 
replacing them with their normal-curve equivalents (called normal deviates). 
Because the normal deviates have a range from -1 to +1, these are often 
subjected to a further transformation so as to eliminate negative values. 
In the Angoff and Ford procedure, the normal deviates are transformed to 
delta-values by the linear transformation, A = + 13. 

It should be noted that item difficulties computed by the above procedure 
ignore what are , at times, important factors. One factor ignored is that 
related to the position of an item in a test. Items occurring near the end 



-19- 



of a test will have fewer correct responses merely because fewer examinees 
attempted them. The greater the importance of time in the test (i.e., the 
higher the "speededness'' of the test) , the greater the importance of this 
item position factor. In the test battery used for the National Longitudi- 
nal Study only the Mosaic Comparisons Test is highly speeded. For this 
reason, only the first 20 items of the second of three parts of the Mosaic 
Comparisons Test were used in the anlaysis. No item analysis procedure, 
however, precludes the necessity of subjective information with which to 
judge the validity of the statistical computations. Accordingly, item 
analyses reported in this study are accompanied by descriptive information 
concerning the item, including its position in the test or subtest, the 
proportion of persons for each group answering the item, and the item itself, 
where test security limitations permit. 

Having computed the item deltas for each item and for each of the ten 
mutually exclusive groups given above, the next step in the procedure was 
to cross-plot the deltas for nine of the groups in contrast to a tenth group 
(North Central White or Caucasian). For each cross-plot, the group being 
examined was placed on the abscissa and the delta values for the North 
Central White group on the ordinate. These cross-plots normally result in 
a narrow elliptical pattern with the major axis extending from the lower 
left to the upper right, very much in the same way as scatter-plots of pre- 
dictive single observations used in correlational analysis. One should note, 
however, that the points in the cross-plots under discussion represent large 
numbers of observations (each point represents two delta values and each 
delta value was determined from the total of all encounters witH the item 
by the group it represents). Thus it is not unusual to find that when a 
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correlational analysis is performed on points so derived, that the corre- 
lational values obtained are typically as high as .98 or .99. Where the 
two groups being compared are very different the correlations will not be 
quite so high. In the present analysis, the line of best fit used was the 
major axis of the elliptical patterns of points rather than the least squares 
regr^'ssion line usually involved in correlational analysis. 

An important feature of the major axis is that it indicates the general 
degree of difficulty of all items \aken together for a given group. If all 
items, on the average, are of equal difficulty for both groups being con- 
trasted, then the major axis will have a slope of 1.0 and an intercept 
(projected) of zero. If the items in general (that is, the whole test) 
are more difficult for the group plotted on the abscissa, then the regres- 
sion line will have an intercept less than zero. And if a test is easier 
for the group whose delta values are plotted on the abscissa, the intercept 
will be greater than zero. The slopes of these inore and less difficult 
lines may be different from l.C, 

Cross-culturally unstable items are those with the most abierrancy 
around the line of best fit for a particular group. Using the symbols x 
and y to represent the delta values for two groups being contrasted and 
using the slope-intercept form of representation for a straight line, 
y = ax + b, and where the constants a and b are determined as 



a = — ^ " 

2r s s 
xy X y 



and b = My - aM^, 



the major axis of the ellipse representing any contrast is defined, 



The 
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symbols M and s above represent the mean and standard deviation, respectively, 
r the correlation between the deltas for the two groups, and the subscripts 
X and y the abscissal and ordinal coordinates. The perpendicular distance, 
of each point, i, in the cross-plot to this majoV axis is given as 

ax. - y, + b ' 
^ /a^ + 1 

When d_j^ = 0 for a particular item, then the item is perfectly stable with 
respect to the two groups being contrasted. When d_j^ > 0, the item tends 
to be more difficult for the group on the ordinate than were most other 
items of the same test for the same group.. Such an item would be said to 
be positively unstable. If < 0, then the item tends to be more diffi- 
cult for the group on the abscissa than were most other items of the same 
test for the same group. Of course, extremely small deviations from zero 
are of little practical significance and could be due to random fluctua- 
tions about the regression line. When a nuinber of groups are being compared 
to some common group\ a vector of d^'s results since there will be one d^ 
for each group. 
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Classlflcation Variables 

Classification variables were used to categorize participants as well 
as to indicate fundamental differences amoug groups being compared. These 
variables were obtained f rom NLS survey questionnaires as follows: 

Sex. Sex of participants was obtained from the NLS Student Question- 
naire. Those survey students who did not respond to this item or did not 
return a Student Questionnaire could not, of course, be classified with 
respect to sex. 

Ethnicity . The ethnicity of participants was obtained from item 84 
of the NLS Student Questionnaire (reproduced below) . All participants who 
omitted the item or did not return the Student Questionnaire or did not 
take the NLS test battery were excluded from the study. 

84. How do you describe yourself? 

(Circle one.) 

American Indian 1 

Black or Afro-American or Negro 2 

Mexican-American or Chicano 3 

Puerto Rican 4 

Other Latin- American origin 5 

Oriental or Asian-American 6 

White or Caucasian 

Other 8 

Language Spoken In Home . The language spoken most often in the home 
(English or not English) was obtained from item 88 of the NLS Student 
Questionnaire as indicated below. 



88. Is English the language spoken most often in your home? 



(Circle one.) 



No 

Yes ^ 
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Time in Community , Time in community was obtained from item 89 of the 
NLS Student Questionnaire as indicated below* 



was obtained from item 90 of the NLS Student Questionnaire, duplicated below. 



89. How long have you ilved in the community in which you how live? 

(Circle one.) 



Three to four years 



Parents ^ Educational Level. Father's education and another's education 



90. What was the highest educational level each of the following persons completed? If you are not sure 
please give your best guess. 

(Circle one number In each column.) 

Father or Mother or Oldest 

male female brother or 

guardian guardian sister 

Doesn't apply 1 1 ^ 

Did not complete high (secondary) school 2 2 2 

Finished high school or equivalent 3 . . . .3 3 

Adult education program 4 4 4 

Business or trade school ...5 5 5 

Some college 6 6 6 

Finished college (four years) 7 . . .7 7 

Attended graduate or professional school (for example, law or 
medical school), but did not attain a graduate or professional 

degree 8 .... 8 8 

Obtained a graduate or professional degree (for example, M.A., 

Ph.D., or M.D.) 9 9 9 

31) 
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Parents^ Income . Parents' income was obtained from item 93 of the NLS 
Student Questionnaire as indicated below. 

93. What is the approximate Income before taxes of your parents (or guardian)? Jncluda taxable and non* 
taxable income from all sources. 

(Circle one.) 



Less than $3,000 a year (about $60 a week or less) 01 

Between $3,000 and $5,999 a year (from $60 to 

$119 a week) 02 

Between $6,000 and $7,499 a year (from $120 to 

$149 a week) 03 

Between $7,500 and $8,999 a year (from $150 to 

$179 a week) 04 

Between $9,000 and $10,499 a year (from $180 to 

$209 a week) 05 

Between $10,500 and $11,999 a year (from $210 to 

$239 a week) 06 

Between $12,000 and $13,499 a year (from $240 to 

$269 a week) 07 

Between $13,500 and $14,999 a year (from $270 to 

$299 a week) 08 

Between $15,000 and $18,000 a year (from $300 to 

$359 a week) 09 

$18,000 a year (about $360 a week or more) 10 



CoTnmunity Size . School community size was obtained from item 40 of the 
NLS School Questionnaire. This item, duplicated below, was completed by the 
NLS Survey Administrators in the schools participating in the NLS. 



40. Which of the following best describes the location of this school? 

(Circle one.) 

In a niral or farming community 1 

In a small city or town of fewer than 50,000 people 

that is not a suburb of a larger place 2 

In a medium-sized city (50,000-100,000 people) 3 

In a suburb of a medium-sized city 4 

\lnei large city (100,000-500,000 people) 5 

In a suburb of a large city 6 

In a very large city (over 500,000 people) 7 

In a suburb of a very large city 8 
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Geographic Classifications . The four geographic divisions of the 
United States, as defined by WESTAT Corporation, were used. The states 
included in each division were as follows: 

(1) Northeast (Maine, New Hampshire, Vermont, Massachusetts, Rhode 
Island, Connecticut, New York, New Jersey, and Pennsylvania). 

(2) North Central (Ohio, Indiana, Illinois, Michigan, Wisconsin, 
Minnesota, Iowa, Missouri, North Dakota, South Dakota, Nebraska, 
and Kansas) * 

(3) South (Delaware, Maryland, District of Columbia, Virginia, West 
Virginia, North Carolina, South Carolina, Georgia, Florida, 
Kentucky, Tennessee, Alabama, Mississippi, Arkansas, Louisiana, 
Oklahoma, and Texas). 

(4) West (Montana, Idaho, Wyoming, Colorado, New Mexico, Arizona, 
Utah. Nevada, Washington, Oregon, California, Alaska, and Hawaii). 
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Sample-Description by Groups 

The original sample of 17,726 cases was reduced to 14,828 cases by 
the requirement • that each case to be analyzed have both an NLS Student 
Questionnaire and an NLS Student Test Book answer sheet. The distri- 
bution of these cases by socio-cultural group, sex, and primary 
language spoken in the home (English or other) is given in Table 3, 
For a few of the cases there was no response to the sex and language 
spoken questions and for this reason the figures do not add to the 
totals in Table 3. Within the groups, the distribution of males and 
females appears to be relatively uniform. Major differences occur, 
however, with regard to language spoken in the home. More Puerto Ricans 
(39) said that English was not the primary language spoken in the home 
than said that it was (38). A very large proportion of Other Latins 
(48) reported that English was not the primary language spoken in the 
home as compared to those who said that it was (56) . The same was the 
case for Mexican-Americans (214 not English vs. 262 English), For 
the other groups, only small proportions ;said that English was not 
the primary language spoken in the home. 

Table 4 shows summary data (means and standard deviations) for 
socioeconomic variables, degree of urbanization, and time in community 
for all groups and the total. The White Western group reported the 
highest mean father ' s . education and the Mexican- American group the 
lowest. The same contrast occurred with respect to mother's education 
with White Western highest-r.^and Mexican-American lowest* For parents' 
income, however, Puerto Ricans' were lowest. School community size 
was highest for Puerto Ricans and lowest for American Indians. 
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Table 3 

Distribution of Cases by Socio-Cultural Group 









Home Langiiiage 




Groups 


Males 


Females 


Not 


Primarily 


Total 






English 


English 




American Indian 


91 


85 


24 


151 


178 


Afro— American 


826 


1,051 


157 


1,720 


1,895 


Mexican-American 


251 


235 


214 


263 


491 


Puerto Rican 


39 


40 


39 


38 


79 


Other Latin 


49 


56 


48 


56 


107 


Oriental 


93 


82 


52 


122 


176 


White Northeastem 


1,278 


1,499 


189 


2,603 


2,798 


White North Central 


1,848 


1,717 


224 


3,356 


3,589 


VJhite Southern 


1,793 


1,740 


230 


3,319 


3,557 


White Western 


1,023 


921 


111 


1,843 


1,958 


Total 


7,251 


7,426 


1,288 


13,471 


14,828 
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Table 4 

Summary Data on Socio-Economic Variables, 
Degree of Urbanization, and Time in Community 





Group 


Father's 


Mother s 


Parents ' 


School Com- 


1 line m 




Education 


Education 


Income 


munity Size 


Community 








Means 






American Indian * ' 


3.02 




/. no 


1 0/. 




Black 


2. 87 


3.18 


3.17 


^ . ZD 


o /. /. 
Z . 44 


1^j"> aT\ mm A. vn 

ric Ai I- cill i i d.ri 


2 .56 


2.45 


3.57 


3.57 


1.94 


Puerto Rican 


2.58 


2.56 


2.85 


6.41 


2.79 


Other Latin American 


3.82 


3.50 


4.45 


5.23 


2.94 


Oriental 


4.23 


3.80 


5.89 


4.09 


2.49 


White Northeastern 


4.10 


3.71 


6.15 


3.86 


1.95 


White North Central 


3.98 


3.72 


5.94 


3.41 


1.96 


White Southern 


4.06 


3.68 


5.69 


3.39 


2.26 


White Western 


4.61 


4.10 


6.40 


3.90 


2.39 


Total 


3.94 


3.65 


5.50 


3.70 


2.17 






Stand 


ard Deviations 




American Indian 


1. o5 




2.90 


2.29 


X.JO 


Black 


1.68 


lo73 


2.37 


2.32 


1. J J 


Mexl c an- Ame r ic an 


1.55 


1.28 


2.47 


£• . wlAJ 


1. zo 


Puerto Rican 


1.39 


1.42 


1.61 


1.51 


1.56 


Other Latin American 


2.30 


1.85 


2.75 


2.27 


1.58 


Oriental 


2.26 


1.99 


2.90 


2.33 


1.39 


White Northeastern 


2.22 


1.88 


2.73 


2.24 


1.26 


White North Central 


2.21 


1.82 


2.70 


2.46 


1.28 


White Southern 


2.33 


1.91 


2.89 


2.33 


1.44 


White Western 


2.32 


1.95 


2.72 


2.39 


1.37 


Total 


2.25 


1.88 


2.91 


2.38 


1.39 
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While these groups differ considerably on variables, such as SES, 
which are well known to be related to test performance, no adjust- 
ments are made for these differences in the present study. Since 
each of the socio-cultural groups were selected by a carefully con- 
ducted random sampling, these group differences are considered to be 
representative of cultural differences. From this point of view, no 
adjustments are appropriate. 
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Re suits 

Cross-plots of all item deltas for each socio-cultural group, in con- 
trast to the White North Central group, are shown in Figures 1 through 9. 
A dotted line has been drawn in each figure at 45 degrees to serve as a 
reference line. Items falling on or near this line are of approximately 
equal difficulty for the White North Central group and the group which is 
being compared to it. The solid line passing through the center of the 
cluster of item points (+'s) is the major axis of the ellipse represented 
by these points. If the solid line falls below the dotted line it indi- 
cates that the NLS battery as a whole was more difficult for the group 
whose deltas are on the abscissa. In Figure 1, for example, the position 
of the solid line relative to the dotted line indicates that the NLS battery 
was more difficult for American Indians than for whites living in the North 
Central region of the United States. Figures 6 and 7 suggest that the NLS 
battery was slightly easier for Oriental-Americans and for whites living in 
the Northeastern region of the United States. 

The results of primary interest, however, are not those related to the 
comparative difficulty of the NLS battery as a whole. Rather, the focus is 
oh specific NLS items— especially those items represented by points (-h's) 
at a distance from the solid line in the cross-plots, A striking example 
of such an item appears in Figure 3 in which Puerto Ricans are compared to 
White Americans living in the North Central region. The point, in Figure 3, 
appearing above the dotted line indicates that the item represented by it 
was easier for Puerto Ricans than for those of the North Central comparison 
group. This point represents 79 observations of Puerto Ricans and 3,589 . 
observations of persons in the comparison group. Thus, the normal degree 
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Figure 1 

Cross-plot of Deltas for American Indian 
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Figure 2 

Cross-plot of Deltas for Afro -American 
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Figure 4 

Cross-plot of Deltas for Mexican-American 




^.00 



8 00 10.00 12.00 14.00 16.00 

DELTAS FOR MEX I CflN-flMER I CRN 



ERIC 



41 



Figure 5 

Cross-plot of Deltas for Other Latin- American 
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Figure 6 



Cross-plot of Deltas for Oriental 
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Cross-plot of Deltas for White-Northeastern 
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Cross-plot of Deltas for White-Western 
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of random variation observed in ordinary scatter plots of single observa- 
tions does not occur in the case of Figure 3. When it is considered that 
the mean total test scores for Puerto Ricans was well below the mean score 
for the comparison group (60,1 vs. 87.9), any item that would appear to be 
easier for Puerto Ricans is worthy of special attention. The same phenom- 
enon occurs when Other Latin-Americans are compared to the white North 
Central group (Figure 5) . Three item points fall above the dotted line 
in this cross-plot. 

The distance of an item point from the dotted line xs of less general 
interest than the distance from the solid line. An an imdex of cross- 
cultural instability the perpendicular distance from the major axis of the 
ellipse for a group (the solid line in Figures 1 through 9) is more meaning- 
ful. These distances (d^'s or D-Values) are given for each subtest of the 

NLS battery, for each item, in Tables 5 through 10. These will be consid- 

^ . _ ____ , x 

ered for each subte¥F^T6prarately . — ^ 

Vocabulary . Table 5 presents the values of d^ for each group on each 
item of the NLS Vocabulary subtest. Negative values of indicate that a 
particular item was easier for a group relative to other items in the NLS 
battery and positive values indicate that the item was more difficult for 
that group than were other items in the NLS battery for the same group. 
Table 5 would indicate that Vocabulary items 2, 4, 13, and 14 were rela- 
tively easier for Other Latin-Americans. Moreover, this relative ease 
extends to both Mexican-Americans and Puerto Ricans for items 13 and 14. 
To a slight degree, the same could be said for item 2, but with much less 
confidence. Item 4, however, would appear to be relatively easier only for 
Other Latin-Americans and Puerto Ricans • Considering the entire Vocabulary 
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subtest, the lower marginal values in Table 5 suggest that, in general, 
this subtest was more unstable for Spanish-speaking students than for 
others (note the higher values for the lower marginal standard deviations 
for the three Spanish-speaking groups). Some possible reasons for these 
instabilities will be discussed later. 

Picture-Number . The D-Values for the Picture-Number test are shown 
in Table 6. Note that a number of the items have no D-Values (indicated 
by the asterisks). These were not computed when Delta-Values were missing 
for an item and the Delta-Values were not computed if either: (i) the 
proportion of a given group responding was less than 50 percent or (ii) 
the proportion responding correctly was greater than .95 or less than .05. 
In Table 6, the primary reason for the missing D-Values was a low response 
rate associated with the speededness of the test. The first part of the 
Picture-Number test is represented by the first 15 items and the second 
part by the last 15 items. Because of the speededness factor and because 
the items in the Picture-Number test are relatively homogeneous across 
cultures (i.e., none of the pictures would appear to have a special meaning 
or lack of meaning for any of the groups being analyzed), the specific item 
D-Values are difficult to interpret. When all of the items of the subtest 
are considered together, however, it is noteworthy that this kind of test 
would appear to be easier for minority groups (except Oriental-Americans) 
than the other subtests of the NLS battery. This is indicated by the lower 
marginal means which are substantially negative for five of the minority 
groups. 

Reading . Like the Vocabulary subtest, the Reading subtest reveals 
considerable cross-cultural instability (Table 7). These instabilities 



I J 'x 



occur principally with respect to blacks and Puerto Ricans , as indicated 
by the lower marginal standard deviations. For blacks, an especially 
interesting aspect of the Reading subtest is represented by the first 
five items. All of the first five items appear to have been relatively 
easier for blacks than other items in the NLS battery. These first five 
items all related to a reading passage concerning the problems and pros- 
pects of black television in the United States. Items 15 and 17 appear 
to be easier for Spanish-speaking students. Reasons why these instabil- 
ities occur are considered later . *' 
Letter Groups . The items of this test, like those of the Picture- 
Number test, are relatively homogeneous across cultures since none of the 
items is appreciably different and there is no obvious reason why one item 
might be relatively more difficult or relatively easier for any given socio- 
cultural group. The low values of d^ shown in Table 8 support such general- 
ization. Very few of the d^^s exceed 1.0 and those few might easily be 
attributed to random variation. And the lower marginal means give no indi- 
caLion that the subtest as a whole is relatively ea=,ier for one group "than 
another . 

Mathematics . As might have been expected, the Mathematics subtest was 
relatively easier for Oriental-Americans than other sections of the NLS 
battery (Table 9) . This subtest also appears to have been relatively more 
difficult for American-Indians, blacks, Mexican-Americans, and Puerto Ricans. 
And while some of the items of the Mathematics test show interesting patterns 
(item 5 relatively more difficult for five minority groups), the content of 
such items gives no indication as to why this might be the case. Ttem 5, 
for example, compared the square root of a number to the same number without 
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the square root symbol. The task was to indicate whether the square root 
was greater, less, equal, or that the relationship could not be determined 
from the information given. Ifliile it could be that minorities receive 
especially poor training in taking square roots, it is not clear why this 
particular item is relatively more difficult than other, similar, mathe- 
matical items. Accordingly, one should be cautious in making any general- 
ization from extreme D-Values when there is no contextual evidence to 
support the generalization. Although the Mathematics subtest and some 
of the iteiuS in it are unstable, one should also be cautious in making 
the inference that such tests are cross-culturally "unfair." 

Mosaic C o mparisons . Since only a middle group of twenty items from 
the NLS Mosaic Comparisons test were analyzed (the first twenty items of 
the second of three parts of the test), it is not appropriate to discuss 
these twenty items as representing the NLS battery Mosaic Comparisons test. 
The results do suggest, however, that this kind of test item is relatively 
stable across cultures (Table 10) . Table 10 suggests as well that the 
primary source of instability is the speededness of the test and that blacks 
and Puerto Ricans were most affected by this speed factor (note the increas- 
ingly large positive D-Values with item number and the low response rates 
for the later items indicated by the asterisks) . Thus it is probable that 
the instabilities which du uccur are not due to the item type, but to the 
fact that speeded tests are relatively ipore difficult for these groups than 
non-speeded tests. 
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Dis cuss ion 

It is not surprising that Vocabulary, Reading, and Mathematics test 
items are less cross-culturally stable than are items such as those in the 
NLS Picture-Number, Letter Groups, and Mosaic Comparisons tests. Language 
is one of the most common differences found across different cultural groups. 
And the penchant for mathematics among Oriental cultures is well known. 
Cross-cultural instability of itself, however, is not reason enough for 
the exclusion of items from a test. The use of an item, or a test within 
which the item is used, as well as the validity of the item need also to 
be considered. In a test used for the selection of persons for advanced 
training in mathematics, for example, it would clearly seem unwise to 
exclude items merely because Orientals performed better on them. Such 
an exclusion would probably attenuate the predictive validity of the test. 

Nor would it seem entirely appropriate to exclude a set of items 
about Black television because these items were relatively easier for 
Blacks. Perhaps all tests need to have more such items and, additionally, 
items associated with ether cultures. The decision of excluding or includ- 
ing specific items in a test is, therefore, a decision far removed from 
any mechanical procedure. Not only do matters of content and predictive 
validity need to be considered, but also matters of subtle cross-cultural 
• differences and the socio-political context that accompanies them. An 
index of cross-cultural stability across socio-cultural groups, however, 
is useful when analyzing the suitability of an item for a particular test. 
Like a speedometer, a thermometer, or any other measuring device, it pro- 
vides information that can help in making decisions. But it is only one of 
several situational variables describing the context in which the decision 
is to be made . 
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The instabilities in the NLS Vocabulary subtest have some especially 
interesting potentialities with respect to the groups speaking Spanish as 
a native language • Although a thorough analysis of this possible socio- 
linguistic phenomenon is beyond the scope of this study, some interesting 
directions for research can be pointed out. 

Items 2, 4, 13, and 14 of the Vocabulary test are of special interest. 
These are items that appeared to be relatively easier for at least some of 
the Spanish-speaking groups than were other items in the NLS battery. Item 
2 asked for the synonym for the English verb, '^convalesce. The correct 
response was the English verb, "recuperate." What is notable about this 
particular kind of item is that these sarqe verbs have cognates in Spanish— 
"convalecer" and "recuperar," respectively. It is thus not surprising that 
Spanish-speakers did better on the item than on other items in the NLS bat- 
tery. What is_ surprising is that one Spanish-speaking group. Other Latin- 
Americans, appears to have done better with item 2 than either the White 
North Central group or the \^ite Southern group. Of the 60 Other Latin- 
Americans who attempted the item, 61 percent gave correct responses- This 
compares to 54 percent and 55 percent, respectively, for the White North 
Central. S^^o^P and the White Southern group. (See Appendix E for actual 
response patterns for all groups on all items.) 

Item 13 of the Vocabulary test is similar, since it also involves a 
double cognate (both the stem and the correct option are Spanish cognates). 
The stem of item 13 was the English adjective, "impetuous," and the correct 
option was the English adjective, "impulsive." The corresponding cognates 
in Spanish are "impetuoso" and "impulsivo." Items 4 and 14 involve cognates 
also' buil these are not double ones, since only the stem in each case is a 
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cognate • Apparently, however, this small advantage helps the Spanish- 
speaker • The stem in item 4 was "novice,-* which has the cognate, "novicio/* 
And the stem in item 14 was "enigma," having the identical cognate in 
Spanish, 

Where a vocabulary word for which a synonym is required is a cognate — 
that is, a word with a similar root in the native language — then the item 
may possibly be easier for persons who speak that native language. And 
if the keyed response is also a cognate, the item might be especially easy 
for a non-English-speaking individual- 'P'inally , if these cognates are more 
common in, say, Spanish than they are in English, then it would not be sur- 
prising at all if Spanish-speaking persons did much better with an item 
than native English-speakers. Such an explanation might offer a reason 
for the fact that some of the NLS vocabulary items were considerably easier 
for Puerto Ricans and Other Latin-Americans, while the same phenomenon did 
not occur to a similar degree for Mexican-Americans . Beyond the question 
of the ease or difficulty of items, analyses of the nature of response 
patterns using techniques like those of Flaugher and Pike (1970) might 
also reveal interesting cross-^cultural phenomena. 
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Conclusions 

This study demonstrates that useful indices of the cross-cultural 
stability of test items (as well as tests) can be created • Perhaps further 
efforts will yield better indices than that used herein. The study also 
shows that no purely mechanical nr statistical procedure is sufficient for 
making decisions about the inclusion or exclusion of items from a particu- 
lar test. An instability across socio-cultural groups may reflect a cul- 
tural tradition that deserves recognition. In such cases, items which 
serve to display cultural differences play a positive role. But in other 
instances, where an instability suggests that the outcomes are due to some 
unintended and undesired consequence, these instabilities serve no useful 
purpose and should be eliminated if possible. A case iti point is where 
Spanish-speaking persons perform better on English vocabulary items because 
the words involved may be cognates more common in Spanish than they are in 
linglish. 

A perhaps more important aspect of the study, however, is that it 
suggests an approach to the present controversy over test bias that might 
be more palatable to minority groups than current approaches. Rather than 
emphasizing predictive analyses based on external criteria (which themselves 
may be biased) , analyses of individual test items in the contexts of their 
use, socio-cultural differences, and other subjective criteria might lead 
to some reconciliation of issues. After having created tests having optimal 
cross-cultural stabilities, without the use of external criteria, the next 
task would be to observe the predictive consequences of such procedures. 
If this new approach yielded- predictive validities not significantly below 
those commonly obtained in less cross-culturally stable tests, then it would 
tend to satisfy psychometricians as well as members of minority groups. 
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Appendices 



The appendices following this page are: 



Appendix A. 
Appendix B. 
Appendix C. 
Appendix D . 

Appendix E. 

Appendix F, 
Appendix G. 



Sample Test Items and Answer Sheet 

r 

Survey Administrators Guide 
Test Analysis 

Item P-Values (proportions of samples . 
responding correctly) 

Item Response Patterns and Statistics 
(including mean total test scores and 
mean subtest scores by group) 

Item Delta-Values (transformed P-Values) 
Item D-Values (standardized) 



Symbols used in Appendices D through G: 



AI - ATTiericaii Indian 

AA. - Black or Afro -American or Negro 

MA ~ Mexican- American 

PR ~ Puerto Rican 

OL - Other Latin-American origin 

OR ~ Oriental or Asian-American 

WE ~ White or Caucasian, Northeastern United States 
WC - White or Caucasian, North Central United States 
WS - White or Caucasian, Southern United States 
WW - White or Caucasian, Western United States 

N - Number of cases 
NR - No response 
P - Proportion of sample or subsample responding 
correctly 
MS - Mean subtest score 
MT - Mean Total Test score 
D - Distance, in delta units, from the najor axis of the 
elliptical pattern of points resulting from the cross- 
plot of item delta-values for a group in contrast to 
a standard comparison group. 
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APPKNDIX A 
Sample Test Items 
and 

Answer. Sheet 



68 



ERIC 



FORM UEE 



i 



"1 



> 



1 



1*/ ' 



* ■ », - • - ^-^-x-' * 



r 





V 



Prepared for the ^ 
UNITED STATES OFFICE OF EDUCATION 
BY EDUCATIONAL TESTING SZRVICE a PRINCETON. NEW JERSEY 
SPRING 1972 




ERIC 







OE FORM NO. 2348-5. 2-72 COPYRIGHT © 1962, 1963, 1967. 196«, 1969, 1970 BY EDUCATIONAL TESTING SERVICE. ALL RIGHTS RESERVED. 



The NLS test battery includes items from ETS tests 
which are in current use. 



To maintain the security of tfiese tests, only 
sample questions from each section have been 
included here. 

Qualified researchers may write for a copy of the 
complete test booklet to: 

Dr. Hunter M, Breland 
Educational Testing Service 
Princeton, New Jersey 08540 
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GENERAL DIRECTIONS 



This cesc has six sections. Some sections have more than one pare. During the time 
allowed for each section or pare, you are co^'ork only on ic. The lime limit for each 
section or separately timed part is printed at the beginning of each section or part, 
and the supervisor will tell you when to begin and when to stop. If you finish a section 
or part before time is called, go back and check your work on that section or part 
only. 

Your score on each section will be the number of correct answers minus a percentage 
of the number of incorrect answers. Therefore, it will not be to your advantage to 
guess unless you are able to eliminate one or more of the answer choices. 

Mark all of your answers on the separate answer sheet, as no credit will be given for 
anything written in the test book. Make your marks on the answer sheet heavy and black, 
as in the examples below. 

Sample Answers 
[B][C][D][|] 



Be sure that the entire box is blackened. 

If you wish to change an answer, erase your first mark completely. 

CONTENTS OF TEST BOOK 



Section I 


Vocabulary 


5 minutes 


Section 2 


Picture-Number 

(Two parts of 5 minutes each) 


10 minutes 


Section 3 


Reading 


15 minutes 


Section 4 


Letter Groups 


15 minutes 


Section 5 


Mathematics 


15 minutes 


Section 6 


Mosaic Comparisons 

(Three parts of 3 minutes each) 


9 minutes 


Total 




69 minutes 
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SECTION I 
VOCABULARY 
Time — 5 minutes 



1 



Directions: Each of Che questions below consists of one word followed by five words or phrases. You are to select the 
one word or phrase whose meaning is closest to that of the word in capital letters. 

Sample Question S ample Answer 

CHILLY: 0 [H [H ^ [1] 

(A) lazy 

( B) nice 

(C) dry 

(D) cold 

(E) sunny 

In order to find the correct answer you look at the word chilly and then look for a word below it that has the same or 
almost the same meaning. When you do this, you see that cold is the answer because cold is closest in meaning to the 
word chilly. " 



(This section of the test contained 
15 items similar to the sample above*) 



Ck 




r^- vQU FINISH BEFORE TIME IS CALLED, CHECK YOUR WORK ON THIS SECTION ONLY. 
DO NOT WORK ON ANY OTHER SECTION IN THE BOOK. 
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SECTION 2 
PICTURE- NUMBER 



• Directions: This is a test of your ability to remember picture- number combinations. The section has two parts. In 
each part you will study a page of fifteen pictures with numbers. On a study page the picture-number pairs will looic 
like this: 






After studying the page showing both pictures and numbers, you will be cold to turn to a page showing the pictures in 
a different order. 

Examples: 



1. 




2. 




3. 




On your answer sheet there are ten boxes with numbers above them for each question. One of the numbers will be the 
number that goes with the picture. You are to blacken the box with that number above it. 



12 
□ 


24 

□ 


31 

□ 


44 

□ 


51 

□ 


57 

□ 


65 
□ 


73, 
@ 


77 

□ 


92 
□ 


15 


27 


34 


41 


46 


55 


62 


75 


82 


89 


□ 


□ 


M 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


13 


19 


28 


34 


46 


58 


62 


67 


73 


97 


□ 


□ 


□ 


□ 


M 


□ 


□ 


□ 


□ 


□ 



The number that goes with the picture of a telephone is 73, so for example 1 you would blacken the box with 73 above it. 
For example 2 you would blacken the box with 34 above it. For example 3 you would blacken the box with 46 above it. 



DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO DO SO. 
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SECTION 3 
READING 
Time— 15 minutes 

Directions: Each passage is followed by questions based on us content. After reading a passage, clioose the best 
answer to each qiiesiion and blacken the corresponding space on the answer sheet. Answer all questions following a 
passage on the basis of whnc is siaied or uriplied in that passage. 



SAMPLE ITEM: 



Of all Che forces reshaping the American city, the 
most powerful and insistent are those rooted in changing 
methods of transportation. The changes are so big and 
obvious that it is easy to forget how remarkable they are. 
The streetcar has all but disappeared, the bus is proving 
an inadequate substitute, commuter rail service vvorsens, 
subways get dirtier, and new expressways pour more 
and more automobiles into the center of town. 

If transit riding continues to decline and if automo- 
bile use continues to rise unchecked, hov/ can the vital 
core of the city survive? Many city planners say flatly 
chat it cannot. The only sure way to relieve congestion 
and preserve the unifying core of the city, supporters 
of mass-transit claim, is to get people out of private 
automobiles and into public transit — "to move people 
not vehicles." 

10. The author suggests that the remarkable changes 
in transportation are often overlooked for which 
of the following reasons? 

(A) They have taken place very gradually over the 

years. 

(B) They have proved to be more effective than old 

methods. 

(C) They are so obvious that they are taken for 

granted. 

(D) They have created new problems for city 

planners. 

(E) They have decreased congestion in the cities. 



11. The author n)entions all of the following as methods 
of transportation which have become less popular 
with commuters EXCEPT 

(A) the bus (B) the automobile 
(C) the streetcar (D) subways (E) railroads 

12. The passage is primarily concerned with which of 
the following? 

(A) Various factors influencing the American city 

(B) The disappearance of the streetcar 

(C) The need for faster automobiles 

(D) The growing network of expressways 

(E) The effects of transportation changes on the city 

13. According to the passage, many city planners feel 
that growing use of automobiles rather than public 
transit will result in 

(A) the construction of more and more expressways 

(B) the deterioration of the vital center of the city 

(C) the relief of congestion in the city 

(D) a decrease in commuter rail service 

(E) demands for limitations on the use of auto- 

mobiles 



GO ON TO THE NEXT PAGE. 



(This^ sGc^tipfn^^ .te^s^t:. (SiO^jtained 

5 reading passages with accompanying 
questions similar to the item above.) 
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SECTION 4 
LETTER GROUPS 



Directions: Each question in this section consists of five groups of letters with 
four letters in each group. Four of the groups have a characteristic in common 
which the fifth group does not have. Decide which group is different, and blacken 
the space on the answer sheet that corresponds to the position (A, B, C, D, or E) 
of your choice. 

Note: The common characteristic will not be based on the sounds of groups of letters 
tho shapes of letters, or whether letter combinations form words or parts of words. 



Sample Questions 



A 

1. NOPQ 

2. NLIK 



B 


C 


D 


E 


1 


DEFL 


ABCD 


HIJK 


UVWX 




PLIK 


QLIK 


THIK 


VLIK 


2 



Sample Answers 



ABC 



In sample question 1, the letters in four of the groups are in consecutive alpha- 
betical order, but group DEFL in column B is not; so space B har been marked 
in the sample answers. In sample question 2, four of the groups contain the 
letter L. Letter group THIK in column D is the group that is different, so space D 
has been marked in the sample answers. 

You will have 15 minutes to work on this section. 



DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO DO SO. 



(This section of the test contained 
25 questions similar to the samples 
above,) 



SECTION 5 
MATHEMATICS 



Directions: Each problem in this section consists of two quantities, one placed in Column A 
and one in Column B. You are to compare the two quantities and on the answer sheec blacken 
space 

A if the quantity in Column A is greater; , 

B if the quantity in Column B is greater; 

C if the two quantities are equal; 

D if the size relationship cannot be determined from the information given. 



Sample Questions 
Column A Column B 



Sample Answers 



Example 1. 
Example 2. 



20 per cent of 10 
6x6 



10 per cent of 20 
12 + 12 



[b][130 

rBifcirDi 



Answer C is marked in Example 1 since the quantity in Column A is equal to the quantity in 
Column B. Answer A is marked for Example 2 since the quantity in Column A is greater than 
the quantity in Column B. 

You will have 15 minutes to work on this section. 



DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO DO SO. 



(This section of the test contained 
25 problems following the format 
described above.) 
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SECTION 6 
MOSAIC COMPARISONS 

Directions: This test consists of pairs of mosaics, that is, patterns of squares like those found on tiled 
floors or walls. Each mosaic is made up of a number of partially shaded squares. The mosaics in each 
pair are identical except for one square which differs in shading. Tlie vertical columns of both mosaics 
are labeled A to C, A to D, or A to E according to the number of columns in the mosaic. Your task will 
be to locate, for each pair of mosaics, the column that contains the single square which is shaded dif- 
ferently. Then mark the space on your separate answer sheet that corresponds to the letter at the head 
of that column. 

Sample Question 



Sample Answer 



ABC 



ABC 





In sample question 1, the right-hand and left-hand mosaics are identical except for the center square 
of column B, so answer space B is blackened in the sample answer. 



Sample Questions 





ABODE 




ABC D E 




Sample Answers 

2. 



In sample question 2, the bottom square In column D Is the one that is different, so answer space D 
is blackened in the sample answers. In sample question 3, the second square in column A is the one 
that is different, so answer space A is blackened in the sample answers. 

There are three parts to this test. All the mosaics in a single part are the same size. EXiring the 
three minutes allowed for each part, you are to work on that part only. Do not move ahead to the next 
part until you are told to do so. Remember only one square is different for each pair of mosaics. 



DO NOT TURN THIS ?AGE UNTIL YOU ARE TOLD TO DO SO. 

(This section contained 116 mosaic comparisons, 
divided into 3 parts as follows: 

'Part 1 - 56 mosaics 

Part 2-33 mosaics 

Part 3 -27 mosaics 
116 
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Survey Administrator's Manual 

IMPORTANT 
Please read this Manual as soon as you receive it. 
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NATIONAL LONGITUDINAL STUDY OF THE HIGH SCHOOL CLASS OF 1972 



Conducted by 

Educational Testing Service, Pnnceton Nev/ Jersey, (or (he 
UNlieD STATES OFFICE OF eDUCADOM 
Sprina 1972 



• SAMPLE QUESTIONS IN THIS MANUAL ARE COPYRIGHTED BY EDUCATIONAL TESTING SERVICE. ALL RIGHTS RESERVED. 
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TO THE SURVEY ADMINISTRATOR 

This Manual has been prepared fco help you carry 
out at your school the survey of the National Longi- 
tudinal Study [Shs) of the High School Class of 1972. 
The survey is being conducted for the U. S. Office of 
Education by Educational Testing Service (ets). 

The NLS is described in the folder National Longi- 
tudinal Study of the High School Class of 1972 and the 
leaflet Information for School Administrators, The 
project will begin with a survey of students and 
counselors in 1,200 schools in the United States. The 
NLS needs the participation of 18 students and 2 
counselors (where possible) in each of these schools. 
The students will require approximately two and 
one-half hours to complete a questionnaire and a 
short battery of tests. Counselors will soend 30 to 
40 minutes filling out a questionnaire about guidance 
activities. 

The cooperation of the students and counselors is 
crucial to the success of the nls. However, you, as 
the Survey Administrator, have an even more critical 
role. In addition to answering questions about the 
school and providing information drawn from the 
school record of each student invited to participate, 
you must see that all tasks are carried out so that the 
survey at your school yields the data it is intended 
to yield. 

This Manual explains your several functions and 
suggests ways to carry them out efficiently. If, after 
reading it, you have any questions, please telephone 
or cable ets. Instructions for communicating with 
NLS staff at ETS are given below. 

Communicating with NLS Staff at ETS 

Please alert the nls staff at ets about any serious 
problem or question you may have about the survey. 
Call (609) 921-9000 collect from schools in the con- 
tinental U. S. and ask for the nls Project or cable 
from Hawaii (eductestsvc). Normal business hours 
are 8:30 a.m. to 4.45 p.m. Eastern Time. 

Written communications and shipments of survey 
materials should be addressed to Educational Test- 
ing Service, P.O. Box 2608, Princeton, New Jersey 
08540. If you spend any money for a telephonccall,^^. 
mailing, or shipment in connection with the survey, 
send a letter giving the amount and reason for the ex- 
penditure to the^above address. You will be reim- 
bursed. 

Preparations fo Date 

Several steps have already been taken by ets, by 
your principal, and probably also by you prior to 
your receiving this Manual. If there are any book- 
lets, memoranda, or other communications about the 
NLS you haven't already seen, be sure to read them 
and confirm that all of the following actions have 
been taken. 



1. The NLS publications previously mentioned were 
received by your principal in early March. 

2. By the first week of March, your principal (or 
you) sent to p:ts a Principal's Reply form identi- 
fying you as Survey Administrator and sjiucifying 
preferred administration and makeup (backup) 
dates between April 4 and April 21. 

3. Your principal (or you) sent to ets a roster of 
your students in Grade 12 (or equivalent) and a 
roster of your staff who are assigned twelfth grade 
counseling duties. 

4. About mid-March, ets mailed an Information Kit 
of NLS materials to your principal with the request 
that it be forwarded to you. The Information Kit 
contains samples of the questionnaires, test book, 
and answer sheet. 

5. A few days ago, you should have received a 
Sample Roster containing the names of students 
and counselors. selected to participate in the sur- 
vey and 30 copies of the Student Invitation folder. 

If any of the above information was not supplied, 
telephone ets immediately to see whether action 
should be taken to correct the omission. Also, if after 
reading this Manual, you conclude that you are not 
able to meet the makeup date previously specified, 
promptly notify ets by telephone of the date you 
prefer. 

Overview of Survey Administrator's Activities 
in April and May 1972 

The following list outlines the actions you will be 
required to take during the course of the survey. 

1. Become familiar with the aims and procedures 
of the survey. 

2. Arrange for the release of participating students 
from classes so they can attend questionnaire 
and test sessions. 

3. Receive the Sample Roster and Student Invita- 
tion folders; invite students and counsd^/rs 
named by ets to participate in the study. 

'^4. Receive shipment of nls questionnaires and test 
materials. Return Materials Receipt Acknowl- 
edgment postcard. 

5. Supply questionnair.es to participating counse- 
lors; obtain their completed questionnaires. 

6. Complete (perhaps with the help of the principal 
or office staff) the School Questionnaire. 

7. Administer the Student Questionnaire and the 
test battery to participating students; 

8. If necessary, arrange makeup session(s) for stu- 
dents to take the tests and complete the ques- 
tionnaire. 
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9. Complete frKThaps with office staff help) a Stu- 
df^nt's Sg}>ioo1 llecord Information form for each 
student invited to participate in the study. 

10. Promptly return all completed questionnaires 
and test answer sheets to ets for processing. 

. 11. If requested by et.s, supply or ask counselors or 
students to supply omitted data; forward these 
data to ETS. 

12. If requested by ETS, arrange for an ETS represen- 
tative to visit your school to appraise the valid- 
ity of the data; assist the visitor as required. 

Each of the above functions except the first is de- 
scribed more fully in the sections that follow. 

Administrators Checklist 

On pages 16-17 of this Manual you will find a de- 
tailed schedule of Survey Administrator's tasks. A 
space is provided opposite each item for checking off 
each task when it is completed. It will facilitate your 
work if you review pages 16-17 frequentlj' and keep 
the record up-to-date. 



PRELIMINARY ARRANGEMENTS WITH 
PRINCIPAL AND STAFF 

Once your survey materials arrive, you must proceed 
immediately with, the detailed plans and specific 
arrangements for all of the tasks that are to be 
performed. 

Review the dates sent to ets on the Principal's 
Reply form. Be certain that all data collection at 
your school can be completed by the makeup date. 

The most efficient way to handle the survey at 
your school would be to excuse participating students 
from classes and other duties for a tliree-houx period 
during which the tests would be administered (while 
the students are fresh) before the questionnaire. 
Total test administration time is 105 minutes, in- 
cluding 10 nimutes for distributing materials and 
reading initial instructions, 16 minutes for reading 
instructions for individual tests, five minutes for col- 
lecting answer sheets, and a five-minute rest at about 
the halfway point. The actual testing time for the 
complete battery is 69 minutes. The time required 
for completing the Student Questionnaire ranges 
from 35 to 60 minutes. 

If participating students cannot be released for a 
three-hour period, schedule two shorter periods on 
consecutive days. Administer the Student Question- 
naire on the first day and the tests on the second. 
Establish the necessary** procedures (for example, the 
issuing of passes to participating students). ^ 

You must be able to give definite arrangements 
and procedures to students af: the time you invite 
them to participate in the nls project (page 7 of 
this Manual). 



You should also arrange for a room for the test and 
questionnaire administrations. Make sure that phy- 
sical factors in the room, suelv'as lighting, heating, 
and ventilating, will be regulated so that the stu- 
dents are comfortable and able to give full attention 
to the tests or questionnaiire. The room should be in a 
location free from outside disturbance. The student 
should have both hands free to deal with a test book 
and answer sheet. If possible, each should have 
enough room to read the book and mark the answer 
sheet without having to pick up or shift either one. 
Most auditoriums are not suitable testing rooms be- 
cause of deficiencies in the factors mentioned above. 

You may or may not need help from the principal 
or the school office staff with the School Question- 
naire (page 7) and the Student's School Record In- 
formation forms fpage 7). You will be able to judge 
your needs after reviewing these materials and the 
pertinent school records. Make arrangements 
promptly for any help you do need. 

Receiving Survey Materials 

This Manual should have reached you with the main 
shipment of survey materials. Promptly check its 
contents against the Survey Administrator's Control 
Sheet. If you find any discrepancy, telephone or 
cable ETS immediately. Then complete the Materials 
Receipt Acknowledgment postcard by entering all 
required information including the School Code (the 
S.C. number on the Control Sheet) and mail it to 
ETS. You should have already received by first class 
mail a large envelope containing the Sample Roster 
(in duplicate) of students and counselors to be in- 
vited to participate in the survey, 30 copies of the 
Student Invitation folder, and a memorandum en- 
titled "Inviting Students and Counselors to Partici- 
pate in the National Longitudinal Study." If you 
have not received the large envelope within two days 
after arrival of your survey materials, telephone or 
cable ETS. 

Storing Survey Materials 

After checking your survey materials, store them in 
the shipping container in a closet, cupboard, or safe 
to which only you ard authorized persons known to 
you have access. The test battery must be kept se- 
cure to insure accuracy of nls results. All materials 
should be on hand when they are needed. 

Obtaining the Cooperation of 
Counselors and Students 

The names of the students and counselors listed on 
your Sample Roster were selected according to the 
principles of random sampling from the complete 
rosters of students and counselors supplied to ets by 
you or your principal. They are not intended to be 
representative of your school or of your senior class, 
although in many cases they will be. The aggregate 
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sample in all 1,200 participating schools is represen- 
tative of schools, counselors, and students through- 
out the nation. To retain the representative quality 
of the sample, it is vital that the students and 
counselors listed on your Sample Roster participate 
in the study. The importance of the n'ls to today's 
youth and to the educational sy.stem and the unique 
contribution each individual can make should be 
impressed upon all those invited to participate. 

Although participation in the study is entirely 
voluntary, it is important that the largest possible 
number of the selected students and counselors ac- 
cept the NLS invitation. One of your primary func- 
tions as Survey Administrator will be to present the 
case for participation and obtain the cooperation of 
those invited students and counselors who seem re- 
luctant to become involved in the project. If despite 
your best efforts you encounter unusual difhculty in 
obtaining the cooperation of these students or coun- 
selors, telephone HTS. 

Inviting Counselors and Students to Participate 

Immediately after you examine the Sample Roster, 
invite the listed counselors and students to take part 
in the survey. The memo that accompanied your 
Sample Roster contains detailed instructions for 
extending these invitations. You will probably wish 
to meet with the counselors to give them an oppor- 
tunity to discuss the survey with you. If they wish, 
counselors may review materials received from ets. 
Distribute a Student Invitation folder to each stu- 
dent on the Sample Roster to supplement your own 
remarks about the importance of participating in the 
project. Let students know that copies of the ques- 
tionnaire are available for inspection at school and 
at home and provide copies for this purpose to stu- 
dents who ask for them. On the Sann:)le Roster, re- 
cord dates of invitation, acceptance, student receipt 
of the questionnaire, and test administration. (Keep 
the two copies of the Sample Roster together so that 
both will show any notes you make on the top copy.) 

At the time a student accepts the invitation to 
participate in the nls, repeat the time and place of 
the scheduled questionnaire and test sessioins) and 
explain any arrangements you have made for stu- 
dents to be excused from classes in order to attend. 
Tell each student to bring his social security number, 
and several No. pencils to the test .session and these 
items and his driver's license number to the ques- 
tionnaire administration. 



COMPLETING THE SCHOOL 
QUESTIONNAIRE 

The School Questionnaire asks questions about your 
-r-school's locale, enrollment, ethnic makeup, staft*, ser- 
vices, facilities, practices, and programs, and also 



about certain kinds of student results. Taken to- 
gether, the completed questionnaires of the partici- 
pating schools will yield a profile of the American 
secondary school. 

Ail, or at least most, of the information required 
for the School Questionnaire lias probably, been 
compiled by the principal or another administrative 
staff member. The amount of help you need in com- 
pleting the School Questionnaire will depend upon 
the availability of such data. Work on the School 
Questionnaire should begin immediately. 



COMPLETING STUDENT'S 

SCHOOL RECORD INFORMATION FORMS 



A Student's School Record Informr-bion form will be 
required for each student invited to participate in the 
study whether or not he actually accepts. (All data 
supplied on these forms will be held confidential, as 
explained below.) Work on these forms should begin 
as soon as you receive your Sample Roster. The in- 
formation asked for can be drawn from a student's 
school records either by you or by a member of the 
school office staff. If several people work at this task, 
you must make sure that all of them follow the .same 
procedures for researching and recording data. Re- 
view each form for completeness. Note on the Sample 
Roster the completion of each of these forms. 



ADMINISTERING THE COUNSELOR 
AND STUDENT QUESTIONNAIRES 

Confidentiality 

Ttnsamportant that completed Student and Counse- 
lor Questionnaires be examined by no one except 
selected ets data-processing personnel. Therefore, 
ETS has provided a Confidential Questionnaire Re- 
turn Envelope for each participant. These envelopes 
are marked to be opened only by educational 

TESTING SERVICE NLS PROJ'^CT MATERIAL.S CONTROL. 

After the participants complete the questionnaires, 
be certain that they place them in these envelopes 
and return th?m sealed to you. You, in turn, will re- 
turn them sealed to ets. 

After initial check-in at ETS, questionnaires will be 
identified by number only. One name-number identi- 
fication file will be prepared and held in secure stor- 
age. The file will contain the names and numbers of 
only those students who complete the questionnaires 
or take the tests. Names and numbers of students 
who do not accept the nls invitation will not appear 
in this file. The file will be used only for adding in- 
formation to the main data file. There will be no pos- 
sibility of associating any person's name with any 
' subgroup or with any item of information. 
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Counselor Questionnaires 

Prepare a questionnaire envelope set for each par- 
ticipating counselor by entering the School Name, 
School Code, and Counselor Number from the Sam- 
ple Roster on the cover of the Counrfclor Question- 
naire and the School Code and Counselor Number on 
a return envelope. Give the appropriate set to each 
counselor. These questionnaires are self-adminis- 
tered. Ask the counselors to return the completed 
questionnaires to you in the sealed envelopes within 
three days. 

Student Questionnaires 

Prepare a question^iaire envelope set for each stu- 
dent who has agreed to participate. Enter the Stu- 
dent Name, Student Number, and your School 
Code from your Sample Roster on the cover of the 
questionnaire. Enter the School Code and Student 
Number on page 1 and also on the return envelope. 

Have several No. 2 pencils and erasers and a pencil 
sharpener at the questionnaire session. At the session, 
distribute tKe appropriate questionnaire, envelope 
sets to the students. The questionnaires are, for the 
most part, self-administered, but you should read the 
directions on page 14 of this Manual to the students, 
answer any questions they may have, and monitor 
the room to maintain order and insure best results. 
Urge the students to be alert to the routing directions 
in the later sections of the questionnaire. 

Questionnaire Makeup Session 

If any students missed the questionnaire administra- 
tion, arrange for them to attend a makeup session. If 
a student cannot attend this session, he may com- 
plete the questionnaire before the makeup date. If he 
is likely to need help in completing the questionnaire, 
arrange to be present when he fills it out. 

GENERAL INSTRUCTIONS 
FOR GIVING THE TESTS 

Before the testing session, study the Timetable for 
Administering Tests and Questionnaires on page 15. 
Prepare an answer sheet for each student by entering 
the School Code, Student Name, School Name, and 
Student Number. 

Supplies You Will Need 

^A^en you administer the tests, you. should provide 
the following: 

—A reliable watch (not a stop watch or any other 
mechanical timing device). 

—A clock (alarm clock size or larger), if possible, in 
the event that there is no clock in the testing room. 



(There should always be two timepieces in the 
room as a check to prevent mistiming.) 

—Several No. 2 pencils and erasers and a pencil 
sharpener. ' 

Seating the Students 

Please follow these guidelines: 

—Seat the students randomly as they enter the 
room. Do not allow them to select their own seats. 

—In a classroom, seat students in alternate rows; 
if space permits, leave every other seat vacant. 

—In a cafeteria or library, seat students so that they 
are at least five feet apart. Candidates should al- 
ways be seated so they face the same direction. 

—Seat left-handed students one behind another in 
a separate row or in the last seat of each row of 
right-handed students. 

—If chairs with right-hand tablet arms are used, a 
left-handed student should be seated so that there 
is a vacant chair .to the left for his use. 

Regulations in the Testing Room 

PROHIBITION OF BOOKS, RULERS, AND OTHER AIDS: 

The students should have nothing on their desks ex- 
cept their test books, answer sheets, and several No. 
2 pencils; they may not use text books, notes, dic- 
tionaries, rulers, compasses, protractors, slide rules, 
or other aids of any kind. 

ROUTINE absences: Routine absences to go to the 
rest room— unlike other absences that will be dis- 
cussed below- need not be noted in your survey 
records. No extra testing time may be allowed for a 
routine absence during a timed test period, and two 
or more students should not leave the room at the 
same time. Collect the test book and answer sheet 
from any student permitted to leave the room. Re- 
turn the same test materials to him upon his return. 

Problems You May Encounter In Giving the Tests 

If any of the following problems occurs in connection 
with the tests, it should be reported on the Problem 
Incident Sheet included in this Manual. If you need 
more space than is provided, attach additional sheets 
to it. Be sure to indicate the test section in which any 
problem occurred and to fill in the identifying infor- 
mation. The Problem Incident Sheet will warn ets 
of incidents that might affect the data. If you experi- 
ence no problems in administering the tests, write 
NONE on the Problem Incident Sheet and fill in the 
identifying information. Return the sheet to ets 
with the completed questionnaires and answer sheets. 
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GROUP MiSTiMiNC:: Report all mistimings. Correct 
any undertiming before you dismiss the students. On 
receipt of a mistiming report, ets will decide if an 
overall adjustment of scores should be made. 

emergencies: Emergencies such as power failure, 
fire, or any other event that distracts the students 
should be reported on the Problem Incident Sheet. 
If, in yoar opinion, the condition isMikely to ad- 
versely affect student performance, move the stu- 
dents to another testing place. Students should not 
speak to one another during the move if it occurs 
while the test administration Is in progress. If you 
are not able to continue satisfactorily in the original 
location or in another location, halt the administra- 
tion, and schedule a makeup session. Telephone ets 
about any problems or uncertainties regarding the 
resumption of testing. 

DEFECTIVE TEST MATERi.ALS: If a student has a defec- 
tive test book, you should collect it, give him a new 
test book, and direct him to continue working on his 
original answer sheet. On the cover of the defective 
test book, print the words defective material and 
indicate the nature and location of the error and your 
school code. Return the defective test book in the 
shipment to ets after the test administration. If a 
student indicates he has a defective answer sheet, 
give him a new one and direct him to write only his 
name on it and continue working with it, starting 
with the next question or the one he stopped working 
on. Report all such instances on the Problem Inci- 
dent Sheet. 

student mistiming: If you find a student working 
on a wrong section of the test, instruct him to pro- 
ceed to the correct section. Record the identifying 
information for the student and enter: 

Worked minutes on section of 

test; missed the time on section . 

absence due to illness: If a student becomes ill 
and must leave the room during the test, collect his 
test book and ah^'vVer sheet. 

If he is able to return and continue testing, give 
him the same test materials. If he has missed a sub- 
stantial portion of the testing, you may prefer to 
have him report to the makeup administration and 
work on those sections of the test he missed, at the 
time the other students take those sections. If a stu- 
dent is unable to return to the test administration, 
notify him to report to the makeup session to take 
the test sections he missed. 

In any case, record on the Problem Incident Sheet 
the identifying information and the test sections that 
are incomplete because of his illness. Enter: 

"Left room after minutes of testing. 

Resumed testing on or at (date or time) ." 
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OTHER problems: A student may mistakenly mark 
his answers on the wrong section of his answer sheet 
or in his test book. All such cases reported or detected 
should be entered on the Problem Incident Sheet by 
recording the student's identifying information and a 
brief explanation. Attach the test book (if it is 
marked) to the Problem Incident Sheet for return to 

ETS. 

Collecting Test Materials 

At the end of the testing session, first collect the test 
books and then the answer sheets. Do not allow a 
student to examine a test book or answer sheet after 
it has been returned to you. As you are collecting the 
test books, have the students check their answer 
sheets to make sure that the identifying information 
is correct. 

Before going on to administer the questionnaire or 
to dismiss the students, count all test books (used, 
unused, and defective) and confirm that none is 
missing. 

When all test materials are in your possession, 
please thank the students for participating in the 
nls. Then read the directions for completing the 
questionnaire or dismiss the students. Return the 
test materials to locked storage until they are ready 
to be sent to ets. 

Test Makeup Session 

As soon as the test administration has been com- 
pleted, advise those students who missed the session, 
or who had to leave before the end of the session, of 
the makeup date and obtain their confirmation that 
they will attend. 

The most important objective of the survey is to 
gather complete data from students and counselors 
in your sample. If, to achieve maximum participa- 
tion, you have to schedule a makeup date later than 
the one originally announced, try to schedule it be- 
fore April 21; in any case, telepUone ets promptly. 



ASSEMBLING AND MAILING INSTRUCTIONS 

Survey materials should be assembled, checked, and 
returned to ets as soon as possible but not later than 
five days after the test makeup administration. 
Check all completed Student's School Record Infor- 
mation forms to be certain that a forjiL bas been 
completed for each student on the Sample'Roster. 
Count all answer sheets and all envelopes containing 
Student Questionnaires and Counselor Question- 
naires. Indicate on the Roster the items received 
from each student and counselor. Place all the an- 
swer sheets in the single Answer Sheet Envelope. 

In the carton supplied for returning nls materials 
to ETS, place the following: 



— 1 copy of the Sample Roster 

—All completed Student Questionnaires, each in its 
own return envelope 

—All Counselor Questionnaires, each in its own 
envelope 

— 1 School Questionnaire 

—All completed Student*s School Record Informa- 
tion forms 

—All completed answer sheets in the single Answer 
Sheet Envelope 

—Any defective test books and any test books con- 
taining answers (see page 9) 

—Problem Incident Sheet 

Attach to the carton one of the shipping labels you 
received in your main shipment of survey materials 
and send the carton to ets. Complete the Notifica- 
tion postcard and mail it to ets. 



FOLLOW-UP 

TO OBTAIN MISSING DATA 

The NLS Project Materials Control staff will promptly 
review the returned questionnaires and forms for 
completeness and will ask you to request students 
and counselors to supply any information that ap- 
pears to have been inadvertently omitted. You will 
not be asked to urge participants to supply informa- 
tion they are reluctant to provide. 

The initial shipment of survey materials included 
a sufficient supply of questionnaires for follow-up 
work. If, however, you sliould discover that you need 
more copies of any item, please call ets for additional 
supplies. Prepare a questionnaire envelope set for 



each counselor or student who will supply missing 
data. Obtain their numbers from your copy of the 
Sample Roster. Ask the students and counselors to 
seal their questionnaires in envelo])es. Forward all 
materials containing supj^lementary data to et.S in 
the large (10"x 13") follow-up envelope included with 
the basic shipment, ets v/ill process these question- 
naires in the manner and with the confidentiality 
provisions described on page 7. 



DISPOSING OF SURVEY MATERIALS 

After the test rruakeup administration, burn or shred 
all test books except books to be returned to eTs. If 
it is not feasible to shred or burn waste materials at 
your school, return the 20 test books to ETS. Do not 
send the books Ibefore May 1, 1972. Use the second 
shipping label you received with your survey mate- 
rials on the cartton containing the books. 

Do not dispose of copies of the questionnaires or 
other NLS materials until ETS notifies you that data 
collection has been satisfactorily completed at your 
school. Questipnmaires and nls materials other than 
test books may be disposed of by any convenient 
means. 



VISITS TO SCHOOLS 

In order to confirm the validity of the collected data, 
ETS will visit app)roximately 5 percent of the partici- 
pating schools, dhosen at random, at mutually con- 
venient times ia April or May. If your school is se- 
lected for a review of this kind, you will be notified 
by telephone and requested to help with the arrange- 
ments for the visit. 



10 



87 



DETAILED INSTRUCTIONS FOR 
GIVING THE TESTS 

Your complete schedule for the nciministration of the 
tests follows. Be .sure to read these instructions care- 
fully before you administer the tests. At tl'ie admin- 
istration, read aloud to the students all directions 
in bold face. .Mlow time for the procedure described 
to be carried out. Do not depart from those direc- 
tions or answer a^y question regarding the content 
of the tests. This will in.sure that all participants in 
the survey take the tct;ts under the same conditions. 

When all students have been admitted and seated 
as directed in ''Seating the Students" on page 8 of 
this Manual, distribute an answer sheet to each 
student. After the an.swer sheets have been distrib- 
uted, be certain that each student 

—has the appropriate answer sheet 

—has a No. 2 pencil 

When the students have had time to look at the 
ansyver sheet, tell them the following: 



Each of you will be given a test book. If you do not 
understand all of the directions for each section, 
please raise your hand. Questions will be answered 
between sections but not after work on any one 
section has begun. There will be a five-minutc rest 
halfway through the tests. When you receive your 
test book, read the directions on the back Cover 
and look at nic when you have finished. Do not 
turn your book over or open it until you are told 
to do so. Are there any questions? . . . 



During the administration of the tests, walk about 
the room, to make sure that each student is working 
on the appropriate section or part and marking his 
answers in the appro[>riate area of the answer sheet. 

Exactly five minutes later, say: 
Please stop work. 

Section 2—Picture-Sumber 

The Picture-Number test is divided into two parts 
of five minutes each. Each part is further divided 
into a three-minute segment for study and a two- 
minute segment for answering. During the time 
allowed for any one segment, you may work only 
on that segment. Read the directions on page 5 
silently as I read them aloud. Do not turn the page 
to begin the test until I tell you to do so. 



This is a test of your ability to remember picture- 
number combinations. The section has two parts. 
In each part you will study a page of 15 pictures 
with numbers. On a study page the picture-num- 
ber pairs will look like the examples below. 
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Be sure that for every space or box you fill in, the 
number on the answer sheet corresponds to the 
number of the question you are answering. When 
you fill in the boxes on the answer sheet, darken the 
ENTIRE box. If you change an answer, erase your 
first mark completely; incomplete erasures may be 
read as intended responses. Do not make any stray 
marks on your answer sheet. Remember that dur- 
ing the time allowed for one section or part, you 
may work only on it. Do not go on to any other sec- 
tion or part until you are told to do so. 



Section 1— Vocabulary 

■ ' " '■• '-^'^ 

Find the area labeled Section 1— Vocabulary on 
your answer sheet. In your test book, the section 
number will appear in the upper corner of the page. 
If a small number appears to the right of the sec- 
tion number, it will refer to the part within that 
section. You will have five minutes to w6rk on 
Section 1. Open your test book to page 3, read the 
directions, and begin work. 



After studying the page showing both pictures and 
numbers, you will be told to turn to a page showing 
the pictures in a different order. Look at the follow- 
ing examples. 



1- 



2. 
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On your answor sheet there arc 10 boxes with 
numbers above tlieni for eaeli question. One of the 
numbers will be the number that goes with the 
picture. You arc to blacken the box with that 
number above it. wSoe how tlie examples are 
marked. 



Exactly 15 minutes later, say: 



Please $top work. Close your book and place it on 
top of your answer sheet. 



12 2-4 Jl •4-4 51 57 65 73 77 92 

Examples. I. Q Q Q Q □ □ □ B □ □ 

15 27 J-4 A\ -{6 55 62 75 S2 39 

^•□□onn □□□□□ 

IJ 19 28 ,U -1(1 58 62 67 7J 97 

^•□□□□H □□□□□ 



The number that goes with the picture of a tele- 
phone is 73, so for example I you would blacken the 
box with 73 above it. For example 2 you would 
blacken the box with 31 above it. For example 3 you 
would blacken the box with 46 above it. Are there 
any questions? . . . You will have three minutes to 
study Part 1. Turn to patfe 7, the study page for 
Part 1, and study the picture-number pairs. 



Exactly three minutes later, say: 

You will have two minutes to work on page 9, the 
test page for Part 1. Turn to page 9 and begin work. 



Exactly two minutes later, say: 

Please stop work. You will have three minutes to 
study Part 2. Turn to page 11, the study page for 
Part 2, and study the picture-number pairs. 



Allow the students about five minutes of rest 
time. You may permit them to leave the room. 



A short time before the end of the rest period, 
summon the students back to the room and say: 

Take your seats and get ready to resume work. 

Section 4-Letter Groups 

You will have 15 minutes to work on Section 4— 
Letter Groups. Turn to page 19 in your test book 
and read the directions silently as I read them 
aloud. Do not turn the page to begin the test until 
1 tell you to do so. 

Directions: 

Each question in this section consists of five groups 
of letters with four letters in each group. Four of 
the groups have a characteristic in common which 
the fifth group does not have. Decide which group 
is different and blacken the space on the answer 
sheet that corresponds to the position (A, B, C, D, 
or E) of your choice. 

Note: The common characteristic will NOT be 
based on the sounds of groups of letters, the shapes 
of letters, or whether letter combinations form 
words or parts of words. Look at the sample ques- 
tions and sample answers. 



— Exactly three minutes later say: 

You will have two minutes to work on page 13, the 
test page for Part 2. Turn to page 13 and begin work. 

Exactly two minutes later, say: 
Please stop work. 

Section 3 — Heading 

You will have 15 minutes to work on Section 3— 
Reading, lurn to page 15 in your test book, read 
the directions, and begin work. 



Sample Questions 

A^ C D E^ 

1. NOPQ DEFL ABCD HIJK UVWX 

2. NLIK PLIK QLIK THIK VLIK 

Sample Answers 

2- SESOE 
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In sample question 1, the letters in foui' of the 
groups are in consecutive alphabetical order, but 
group DEFL in column B is not; so space B has been 
marked in the sample answers. In sample question 
2, four of the groups contain the letter L, Letter 
group THIK in column D is the t^roup that is 
diflfcrent, so space D has been marked in the 
sample answers. Are there any questions? . . . You 
will have 15 minutes to work on this section. Turn 
the page and begin work. 



Exactly 15 minutes later, say- 



Answer C is marked in example 1 since the quantity 
in Column A is equal to the quantity in Column B. 
Answer A is marked for example 2 since the quan- 
tity in Column A is greater than the quantity in 
Column B. 

You will have 15 minutes to work on this section. 
Are there any questions? . . . Turn the page and 
hej'in work. 



Exactly 15 minutes later, say: 



Please stop work. 



Please stop work. 



Section 5—^fathe7J^atics 

Turn to the area of your answer sheet labeled 
Section 5-Mathematics. You will have 15 minutes 
to work on the Mathematics test. 

Turn to page 21 in your test book and read the 
directions silently as 1 read them aloud. Do not 
turn the page to begin the test until I tell you to 
do so. " 



Directions: 

Each problem in this section consists of two 
quaVUities, one placed in Column A and one in 
Column B. You are to compare the two quantities 
and on the answer sheet blacken space 

A if the quantity in Column A is greater; 
B if the quantity in Column B is greater; 
C if the two quantities are equal; 
D if the size relationship cannot be determined 
from the information given. 



Section G—Mosaic Comparisons 

Turn to page 25 in your test book and read the 
directions for Section 6-Mosaie Comparisons 
silently as I read them aloud. 



Directions: 

This test consists of pairs of mosaics, that is, 
patterns of squares like those found on tiled floors 
or walls. Each mosaic is made up of a number of 
partially shaded squares. The mosaics in each pair 
are identical except for one square which differs in 
shading. The vertical columns of both mosaics are - 
labeled A to C, A to D, or A to E according to the 
number of columns in the mosaic. Your task will 
be to locate, for each pair of mosaics, the column 
that contains the single square which is shaded 
differently. Then mark the space on your separate 
answer sheet that corresponds to the letter at the 
head of that column. Look at Sample Question 1. 



Look at the sample questions and sample 
answers. 



Sample Question 



Sample Answer 



EXAMPLE 1 
EXAMPLE 2 



Sample Questions 

Column A Column B 

20 percent of 10 10 percent of 20 

6X6 12+12 



ABC ABC 




I. [A] 00 



Sample Answers 



1. 
2. 



SEE 



In sample question 1, the right-hand and . left-' 
hand mosaics are identical except for the center 
square of Column B, so answer space L' is blackened 
in the sample answer. Look at es:^mp'c» 2 and 3. 
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Sample Questions 

abode: abode 




Sample Answers 



DETAILED INSTRUCTIONS FOR 
ADMINISTERING THE 
STUDENT QUESTIONNAIRE 

When all students have been admitted and seated as 
directed in ''Seating the Students" on page 8 of this 
Manual, make certain that each student has the 
copy of the questionnaire and the return envelope 
you have prepared for him. Then read the explana- 
tion exactly as it is printed. 



2- 01BJ0O 

[I][£][I] 



3. 



In sample question 2, the bottom square in Column 
D is the one that is different, so answer spaee D is 
blackened in the sample answers. In sample 
question 3, the second square in column A is the 
one that is different, so answer space A is blackened 
in the sample answers. 

There are three parts to this test. All the mosaics 
in a single part are the same size. During the three 
minutes allowed for each part, you are to work on 
that part only. Do not move ahead to the next part 
until you arc told to do so. Remember only one 
square is different for each pair of mosaics. 



The only right answers on thi^ questionnaire are 
those that reflect your own goals, experiences, and 
attitudes. In no case will the answers of individual 
students be singled out. The results, in the form 
of statistical summaries, will be used for research 
purposes only. If you are uncertain about any 
question, please raise your hand and I will try to 
help you understand what is asked for. Read the 
directions inside the front cover silently as I read 
them aloud. 



SOCIAL SECURITY NUMBER. If you have a social 
security number, enter it in the spaces provided 
on the inside front cover. . . . 



Directions 



Are there any questions? . . . You will have three 
minutes to work on Part 1. Turn to page 27 and 
begin work on Part 1 on your answer sheet. 



• This questionnaire is divided into sections of 
questions. All students are asked to start by 
answering questions in the sections lettered A, 
B, and C. Then you will be asked to follow the 
directions to answer questions in the sections 
that apply to YOUR particular plans for the 
year after you leave high school. 



Exactly three minutes later, say: 

Stop work. You will have three minutes to work 
on Part 2. Turn to page 30 and begin work on 
Part 2 on your answer sheet. 



• Read carefully ALL directions for each question 
you answer. It is important that you follow 
these directions carefully. 

• When you are asked to circle a number, please 
make a heavy black circle. Look at the example. 



Exactly three minutes later, say: 

Stop work. You will have three minutes to work 
on Part 3. Turn to page 32 and begin work on 
Part 3 on your answer sheet. 



Exactly three minu£es later, say: 

Please stop work. 
Collect test materials (see page 9). 



What grade are you in? 

(Circle one.) 

Grade 9 1 

Grade 10 2 

Grade 11 3 

Grade 12 0 

• Circle as many numbers as the directions indi- 
cate for each question you answer. 

• Completely erase any answers you wish to 
change. 
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• When you have completctl the questionnaire, 
put it in the envelope that has been given to you 
and seal the envelope. No one at your school will 
see or read your answers. 

This questionnaire is not a test. You raay omit 
any question that you or your parents would 
consider objectionable. 

On page 1, enter the following: 

SEX. Circle the appropriate number to indicate 
your sex. . . . 

DATE OF BIRTH. In the spaces provided, enter 
the numbers indicating the month, the day, 
and year of your birth. . . . 



Now tell the students to fill out the questionnaire 
on their own. 



TIMETABLE FOR ADIVIINISTERING 
TESTS AND QUESTIONNAIRES 

Listed below is a suggested timetable for admin- 
istering both the tests and the que.stionnaire at a 
single session. Even though you administer the tests 
and the questionnaire on two consecutive days, you 
should find this schedule helpful. Note that when 
both tests and questionnaire are administered in a 
single session, it is advisable to give the tests first; 
but if two separate sessions are necessary, the ques- 
tionnaire should be administered in the first session. 

9:00 a.m. Distribute test materials. 

9:10 a.m. Read instructions for Vocabulary Test. 

9:11 a.m. Begin Vocabulary Test. 

9:16 a.m. Read instructions for Picture-Number 
Test. 

9:18 a.m. Begin Picture-Number Test. 

9:28 a.m. Read instructions for Reading Test. 

9:30 a.m. Begin Reading Test. 

9:45 a.m. Begin five-minute rest break. 

9:50 a.m. Reassemble students. Read instructions 
for Letter Groups Test. 

9:54 a.m. Begin Letter Groups Test. 

10:09 a.m. Read instructions for Mathematics 
Test. 

10:13 a.m. Begin Mathematics Test. 

10:28 a.m. Read instructions for Mosaic 
Comparisons Test. 

10:31 a.m. Begin Mosaic Comparisons Test. 

10:40 a.m. Stop testing. Collect answer sheets and 
test books. 

10:45 a.m. Begin 10-minute rest break. 

10:55 a.m. Reassemble students. Distribute 
Student Questionnaires. 

11:00-12:00 Students complete questionnaires, seal 
them in envelopes, and turn them in. 
(There is no need, for survey purposes, 
for a student to remain after he has 
completed and turned in his question- 
naire.) 
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ADMINISTRATOR'S SCHEDULE OF TASKS 



This section of the Manual is dcsiKned to help you keep track of the tasks you have to carry out to complete 
your school's j)art in the National Longitudinal Study. Tasks that are not self-explanatory have been 
described in the preceding pages. 



Check 
when 

completed NLS Deadline 
Upon appoititment 



Task 

1. Confirm that the following have been compiled and sent lo 
ETS via the Principal's Reply form: 

a. List of your school's students in grade 12 or equivalent 

b. List of staff members who perform twelfth-grade counsel- 
ing functions part- or full-time 

c. Preferred date and makeup date for data collection 



Upon appointment 



2. Read the folder Ncilional Longitudinal Study of the High 
School Class of 7972 and the leafiet Information for School 
Administrators, Review all other materials. 



Upon appointment 

Upon receipt of 
Sample Roster 



... 

3. Review the Information Kit sent to your prmcipal. 

■ tjr' 

4. Invite listed counselors to take part in the survey on the 
appointed date. Distribute invitations to students on the list 
and obtain their agreement to participate. 



Upon receipt of the 
Survey Administrator's 
Control Sheet 



5. Notify the person at your school who usually receives 
packages to expect the nls shipment and to inform you 
when it arrives. 



Upon receipt of the shipment 



6. Check the shipment. If your shipment is incomplete, call 
ETS collect immediately. Complete the Materials Receipt 
Acknowledgment postcard and return it to ETS. Store the 
materials with care. 



As soon as possible 



7. Arrange for a room for the administration of the question- 
naire and tests. 



Upon receipt of 
Sample Roster 



8. With assistance as needed from school office staff, complete 
a Student's School Record Information form for each 
student invited to participate. 



As soon as possible, but not 
later than the specified 
makeup date 



9. Complete the School Questionnaire with assistance as needed 
from principal or school office staiT. 



As soon as possible 



10, Record the number and name of each student participant on 
a Student Questionnaire, envelope, answer sheet, and Stu- 
dent's School Record Information form. Record the identi- 
fying information for each counselor participant on a 
Counselor Questionnaire and envelope. 
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As soon as possible 



11. Distribute qucHtionnaires and cMivclopos ('on which you have 
recorded the apiiroi)rialo identification) to participating 
counselors; and ask theni to return the questionnaires to you 
within three days. 



On specified date 



On specified makeup date 



12. Administer the Student Questionnaire. 

13. If necessary, administer the Student Questionnaire to stu- 
dents who did not attend the earlier questionnaire session. 



On specified date 

On specified makeup date 



14. Give the tests. 

15. If necessary, give the tests to students who did not attend 
the earlier test session. 



After each administration 16. Complete the Problem Incident Sheet. 



Prior to mailing 



After makeup administration 



As soon as possible but not 
later than five days after the 
makeup administration 



As soon as possible but not 
later than five days after the 
makeup administration 

After makeup administration 

When contacted 

If requested 



17. Check all completed Student's School Record. Information 
forms. 

18. Complete the Sample Roster form. Draw a line through the 
names of those counselors and students who did not par- 
ticipate. 

19. Place the following in the carton provided for return of 
materials to ets: 

—Sealed envelopes containing Counselor Questionnaires 
-Sealed envelopes containing Student Question nitres 
--Answer Sheet Envelope containing test answer sheets 
-Student's School Record Information forms 
—School Questionnaire 
—One copy of Sample Roster 
—Problem Incident Sheet 

-Defective test books and test books containing answers to 
test questions (if any) 

20. Mail the carton. Complete the Notificatioi-j^. postcard and 
mail it to ets. 



21. Destroy the test booklets. 

22. Assist ETS in following up missing data. 

23. Arrange for a project staff membQr to visit the school at a 
mutually convenient time in April or May. 



When notified by ets that 
your data collection has 
been completed 



24. Destroy all remaining survey materials. 



O.M.B. No. 51-S-72008 
APPROVAL EXPIRES 1.31-73 



NATIONAL LONGITUDINAL STUDY OF THE HIGH SCHOOL CLASS OF 1972 

PROBLEM INCIDENT SHEET 

(Always include student identification and test section aQected.) v-j 



SCHOOL NAME 

SURVEY ADMINISTRATOR'S SIGNATURE 



.CITY AND STATE 

TODAY'S DATE 



SCHOOL CODE 

OE FORM 2348-13, 3/72 



APPENDIX C 
Test Analysis 
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Test Analysis 

NATIONAL LONGITUDINAL STUDY OF THE HIGH SCHOOL CLASS OF 1972 

UEE 



March, 1973 
SR-73-29 



EDUCATIONAL TESTING SERVICE 

Princtton, New Jersey Berkeley, California 



NATIONAL LONGITUDINAL STUDY OF THE HIGH SCHOOL CLASS. OF 1972 

UEE 



March, 1973 Frances Swineford 

The test battery, Form UEE, that provides part of the data collected for a 
national longitudinal study of the educational and career progress of a carefully 
designed probability sample of 1972 high-school seniors was administered in the 
spring of 1972 to 15,863 students, 15,596 of whom became the final working sample 
for the study. The test outline is as follows: 

1. Vocabulary (5 minutes) 

2. Picture-Number 

Part 1. (study 3 minutes, test 2 minutes) 
Part 2. (study 3 minutes, test 2 minutes) 

3. Reading (15 minutes ) 

4 . Le t ter Groups (15 minutes ) 

5 . Mathematics (15 minutes) 

6. Mosaic Comparisons 

Part 1. (3 minutes) 
Part 2. C3 minutes) 
Part 3. C3 minutes) 

Eleven scores were obtained for each student: a total score on each of 
the six sections and the score on each part of Section 2 and Section 6. 



NOTES ON PRINCIPAL FINDINGS 

Total Group 

15,863 national sample of high-school seniors. 

Sample 

Sample of 1,955 cases slightly more able than total group (but see text)'. 
Appropriateness of Test to Group 

All distributions cover effective score range. Letter Groups discriminates 
better at low end than at high end of score scale. 

Reliability 

Estimates of .784 for Vocabul2f/, .845 for Picture-Number, .797 for Reading, 
• 861 for Letter Groups, .866 for Mathemaxilcs , and, probably, about .90 for Mosaic 
Comparisons. 

9a 
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Speededness 

Vocabulary, Reading, and Mathematics probably not unduly speeded. 
Evidence of some degree of speededness in Letter Groups. Speededness not 
measurable for Picture-Number. Mosaic Comparisons are speed tests, as 
intended. 

Mean Item Difficulty 

Mean deltas of 12.8 for Vocabulary, 12,1 for Reading, 10,4 for Letter 
Groups, and 11,7 for Mathematics. Deltas considered not appropriate for use 
with such tests as Picture-Number and Mosaic Comparisons. Middle-difficulty 
reference values are 12.0 for 5-choice items (Sections 1, 3, 4) and 11.7 for 
4-choice items (Section 5). 

Mean Biserial Correlation 

Means of .63 for Vocabulary, ,58 for Reading, .65 for Letter Groups, 
and .61 for Mathematics. Criteria are corresponding total scores. 



TOTAL -GROUP STATISTICS 

Frequency distributions of nine scores that were obtained for the 
total group are presented on pages A to E, The part scores for Picture-Number 
are not included on these pages, since they were not among the reported scores. 
All the scores extend over a wide range, from less than zero to maxim^jm possible 
values. If a test were of middle difficulty for this group (except for speed 
tests), the mean would approximate one-half of the number of items. Vocabulary 
appears to be somewhat difficult; Letter Groups, quite easy; and Reading and 
Mathematics, near middle difficulty for the group. Picture-Number and the 
Mosaic Comparisons tests are speed tests, whose difficulty can not be judged 
in the same way as that of power tests. 



SAMPLE STATISTICS 



A sytematic sample of 1,955 cases was drawn for detailed analysis. 
For this purpose no record was accepted if one or more sections had been left 
blank for any reason. This restriction is more likely to eliminate low- ability 
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students than the able ones, and the analysis sample is indeed a little more able, 
on the average, than the total group, as is apparent from the score data for the 
power tests, given belox^. Since the total-group scores were recorded to three 
decimal places and the sample scores were rounded to the nearest whole numbers, 
with those ending in .5 rounded to the next higher integer, a value of 0,125 has 
been added to the total-group means for Vocabulary, Reading, and Letter Groups in 
order to make them comparable v/ith the sample means. 

Sample Total Group 

Mean S.D. Mean S.D. 

Vocabulary 6.28 4.22 6.14 4.16 

Reading 9.71 5.15 9.45 5.12 

Letter Groups 16.19 5.96 15.92 6.01 

Mathematics 12.53 7.47 12.25 7.43 

The mean differences for Reading and Letter Groups are statistically 
significant at the 5 per cent level of confidence, but the actual differences are 
considered small from a practical point of view. 

Estimates of the reliability of most of the scores are given at the top 
of page F. Those for Vocabulary, .784; Reading, .797; Letter Groups, .861; and 
Mathematics, .866, were computed by the Kuder-Richardson formula (20) adapted for 
use with R-KW scores. Internal estimates, such as those provided by the 
Kuder-Richardson procedures, arp not appropriate for use with speed tests. The 
best estimate of the reliability of each part of Picture-Number is the correlation 
between them, .726. The reliability of their sum was computed by the formula, 

T . , . T . ^ . T error variance 

reliability = 1 - ; : , 

total variance ' 

where the error variance is the sum of the squared standard errors of measurement 
of the 15-item parts and the total variance is the variance of the 30-item total 
score. The resulting reliability estimate is .345. No attempt has been made to 
iissess the reliability of the three Mosaic Comparison tests, for they differ • 
from one another with respect to both complexity of items and number of items. 
Correlations between any two may well underestimate the reliability of either. 
The reliability of any one of them is probably no lower than .75 and possibly 
much higher. The reliability of their sum is probably about .90. 
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Intercorrelations among all eleven scores are presented in the middle 
portion of page F. The six correlations among the four power tests range from 
.496 between Vocabulary and Letter Groups to .686 between Vocabulary and Reading. 
The correlations among the three parts of Mosaic Comparisons are ,74 between 
Parts 2 and 3, .68 between Parts 1 and 2, and .58 between Parts 1 and 3, which 
are the most dissimilar pair. It is not possible to judge the Picture-Number test 
with respect to its power and speed characteristics. Its correlations with the 
power tests are .292, Vocabulary; .355, Reading; .451, Letter Groups; and .423. 
Mathematics, ar.d its correlations with the three parts of Mosaic Comparisons are 
.318, .364, and .352. Thus, on the average, it is more highly correlated with the 
power scores than with the speed scores, but the difference is extremely small. 

Data relating to speededness are given at the bottom of page F. If 
at least 80 per cent of the group reach the last item and if virtually every 
one reaches at least three-quarters of the items, speed may be considered an 
unimportant element in the score. The dcta for Vocabulary, Reading, and Mathematic 
satisfy the first of these somewhat arbitrary criteria of an unspeeded test, but 
none satisfies the second. There is evidence of some speed in Letter Groups, since 
the last four items were reached by less than 80 per cent of the sample. The data 
for the Mosaic Comparison tests show all to be highly speeded, as intended. The 
Picture-Number tests present two difficulties: first, it is not possible to 
evaluate the timing allotted to the study periods, and, secondly, it can not 
reasonably be assumed that an examinee would record his responses in item-number 
order— on the contrary, it is perhaps more likely that he would first pick out 
the items that he best remembers and then go back to those about which he" is less 
certain. For this reason, it has been decided not to report speed data for this 
tes t . 

Special score data are presented on pages G to 0, which include frequency 
distributions of the number of items answered right, answered wrong, omitted, and 
(except Picture-Number) not reached and two-way dis triba^tions of Score versus R+W. 
An unspeeded test would be expected to have a low NR mean and standard deviation 
and a high proportion of entries in the right-hand coluims of the two-way table. 
A speed test, on the other hand, would have a high NR mean and standard deviation 
and a high proportion of entries along the main diagonal of the two-way table. 
Vocabulary, Reading, and Mathematics (pages G, J, and L, respectively) have the' 
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characteristics of an unspeeded test. In the case of Letter Groups (page K) , the 
NR data are low enough to suggest that the test is not speeded, but the dropping 
out through the last five items, already noted, and the configuration of entries 
near the upper end of the principal diagonal strongly suggest that a substantial 
proportion of the group would have increased their scores if they had been given 
more time. 

The speed tests, Mosaic Comparison (pages M, ri, and 0), clearly 
exhibit the typical speed-test characteristics. Each NR mean exceeds the Score 
mean, and each NR standard deviation differs little from the Score standard 
deviation. The mean number of errors is only 0.87 for Part 1, 1-11 for Part 2, 
and 1.36 for Part 3, the slight increase consistent with the increasing complexity 
of the item type, and the mean number of omissions is even lower: 0.25, 0.12, 
and 0.13. 

The configurations of entries in the two-way tables for the Picture- 
Number tests (pages H and I) resemble the typical speed-test configuration. But 
in this case failure to respond may well be more a function of memory than a 
function of timing. Tne R distributions are quite unlike the R distributions 
for Mosaic Comparisons. Each has its modal value at 15, the maximum possible 
score — a strong hint that speed is not a prominent element. 

If an answer sheet were marked at random, the resulting score would most 
probably approximate zero, and the chances are 99 out of 100 that it would, lie., 
below the dashed line drawn near the bottom of the two-way table. Per cents of 
scores on the several tests that are within the chance area defined in this manner 
are more than 34 for Vocabulary, less than 5 for each part of Picture-Number, 
about 22 for Reading, 6 for Letter Groups, nearly 24 for Mathematics, and no more 
than 5 for any part of Mosaic Comparisons. The right-hand columns of the Mosaic 
Comparisons tables consistently contain scores for a few individuals who succeeded 
in reaching the end with a high degree of accuracy and a handful of individuals 
with scores within the chance area. One wonders whether the latter examinees 
failed to understand the directions, or felt constrained to reach the end without 
due regard for accuracy, or possibly had some visual deficiency that made the 
task particularly difficult for them. 



Item statistics are summarized on page P. At the top of the page 
are frequency distributions of the difficulty index, delta. The numbers in the 
stub of the table indicate the range within which delta usually lies. The 
middle-difficulty value (a useful reference point) varies somewhat with the 
number of options per item. For a 5-choice item, such as those of Sections 1, 
3, and 4, it is about 12.0, and for a 4-choice item (Section 5), it is 11.7. 
Not only is the Vocabulary mean delta as much as 0.8 higher (harder) than middle 
difficulty but also only three items are easier than this reference value, a 
finding that explains the large proportion of Vocabulary scores that are within 
the chance area. 

The Picture-Number test can best be considered as a unit; that is, a 
group of high-school seniors can in two minutes recall with better than 50 per 
cent accuracy a set of 15 such items immediately after three minutes of study. 
For this reason information about individual items has not been included. ^ 

The mean deltas of 12.1 for Reading and 11.7 for Mathematics show 
both tests to be of middle difficulty for this group. The very easy Letter Group 
test has a mean delta of 10.4, which is 1.6 delta points below the middle-difficulty 
reference value. 

Each part of Mosaic Comparisons is a speed test in the sense that 
almost every item answered was answered correctly, and few items were omitted. 
Consequently, item statistics are not reported for these tests. 

At the bottom of page P are distributions of the biserial correlations 
of item scores with criterion scores. The criterion for the items in a column 
is the score on the section indicated at the head of that column. As one might 
expect when the group consists of a grade at high-school level, without selection 
on any basis, these correlations are high. Mean values range from .58 for Reading 
to .65 for Letter Groups. Few coefficients are in the .40's, and only two are in 
the .30's« It should also be noted that there may be a noticeable spurious eifect 
on the correlations for Sections 1 and 3, because each item is a substantial part 
of its own criteriono When there are 25 or more items in a criterion, the 
spurious effect in the correlation between one of the items and the total is 
relatively small and may be ignored for practical purposes. 
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TEST ANALYSIS REPORT FORM 
Test National Longitudinal Study Subject 



Taken by Grade 12 students 



Form UEE 



Date Spring 1972 



Project 825 job 50 



Vocabulary 


Picture -Number 


Raw 

Score 
X 


Standard 
Score 
y 


f ..^ 


Percentile 
Rank of 
Lower Limit 


Y 


Y 


± 


Percentile 
Bank of 
Lower Limit 






of Interval 


A 






of Interval 


15 




342 


97o 8 


30 




J ^ ~j 




14 




575 


94 2 


28 - 29 




n 56 


ft6 7 


13 




720 


89 . 7 


26 - 27 




Q 7 1 


fin 6 


12 




208 


88.4 






Q 90 


7 A ft 


11 




786 








O-UO O 


0 0 , u 


10 




9 32 


77 5 


9n — 91 






An ft 

DU . o 


9 • 




1208 


69.9 


18 - 19 




1166 


53.5 


8 




1318 


61.6 


16 - 17 




1312 


45.2 


7 




622 


57.7 


14 - 15 




1260 


37.3 


6 




1414 


48.8 


12 - 13 




1301 


. 29.1 


5 




1544 


39.0 


10 - 11 




■120J„._ 


^; 21.5 


4 




1597 


29.0 


8 - 9 




1052 


14.9 


3 




1550 


19.2 


6 - 7 




886 


9.3 


2 




639 


15.2 


4 - 5 




681 


5.0 


1 




1046 


8.6 


2 - 3 




492 


1.9 


0 




772 


3.7 


0 - 1 




250 


0.3 


- 1 




385 


1.3 


- 2 - - 1 




47 


0.01 


- 2 




176 


0.2 


- 4 - - 3 




2 


0.00 


- 3 




29 


0.0 






















15863 






















15863 












M = 6.02 


Conversion Data 


M =16.76 


Conversion Data 


X 








X 






0 = 4.16 


No conversion. 




0 = 8.13 


No conversion. 




X 






X 






M = 






M = 






y 






y 






a = 






0 = 






y 






y 






Md = 5.64 






= 16.64 






X 




1 0 -i 


X 






(15 items) 






(30 items) 
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TEST ANALYSIS REPORT FORM 

Test National Longitudi n al Study Subject 

Taken by Grade 12 students 



Forrri UEE 



Date Spring 1972 



Project 825 Job 50 



Reading 



Letter Groups 







Raw 
Score 


Standard 
Score 


f 


■Percentile 
flank of 
jOwer Limit 


Raw j 
Score 
X 


Standard 
Score 
Y 


f 


Percentile 
Rank of 
Lower Limit 


X 


Y 




of Interval 








of Interval 


20 




117 


99. 3 


24 - 25 




981 


93.8 


19 




312 


97.3 


22 - 23 




1792 


82. D 


18 




572 


93.7 


20 - 21 




2547 


66.5 


17 




88 


93.1 


18 - 19 




1 / / o 

244o 


5X . 0 


16 




' 740 


88.5 


16 - 17 




1709 


^0 . J 


15 




1071 


81, 7 


14 - 15 




1 c n o 


jO. 2 


14 




1166 


74.4 


12 - 13 




ilbz 


22 . 9 


13 




12 75 


66 . 3 


10 - 11 




1 A "7 C 

1075 


T a 1 

lb . 1 


12 




318 


64. 3 


8 - 9 




901 


10 . 4 


11 




1167 


57.0 


6 - 7 




493 


7 . 3 


10 




12 75 


48.9 


4 - 5 




526 


4 .0 


9 




1206 


41.3 


2 - 3 




246 


2.5 


8 




1179 


33.9 


0 - 1 




243 


0.9 


7 




401 


31.4 


- 2 - - 1 




121 


0.2 


6 




994 


25.1 


- 4 - - 3 




19 


0.1 


5 




957 


19.1 


- 6 - - 5 




8 


0.0 


4 




802 


14.0 










3 




705 


9.6 






15863 




2 




260 


7.9 - 










1 




504 


4.8 










0 




395 


2.3 










- 1 




209 


0.9 










- 2 




107 


0.3 










' - 3 




19 


0.2 










- 4 




18 


0.04 










- 5 




6 


0.00 














15863 













ERiC 



M = 



a = 



M = 



Md - 





9. 


32 




5. 


12 
















9. 


88 



Conversion Data 



No conversion. 



M - 15.79 



a = 6.01 



Conversion Data 



No conversion. 



M = 



a = 



M = 17.13 



(20 items) 



(25 items) 



TEST ANALYSIS REPORT FORM 

Test National Longitudinal Study Subject 

Taken by Grade 12 students 



Form 



UEE 



Date Spring 1972 



Project 825 job 



50 



Mathematics 



Mosaic Comparisons, Part 1 



Raw 

Score 
X 



24 




25 


22 




23 


20 




21 


18 




19 


16 




17 


14 




15 


12 




13 


10 




11 


8 




9 


6 




7 


4 




5 


2 




3 


0 




1 


- 2 




- 1 


- 4 




- 3 


- 6 




- 5 



Standard 
Score 
Y 



Percentile 

Rank of 
Lower Limit 
of Interval 



1168 
809 

1631 
920 

1609 
949 

1616 

1022 

1582 
982 

1422 
812 
875 
318 
139 
9 

15863 



Raw 
Score 
X 



92.6 
87.5 
77.3 
71.5 
61.3 
55.3 
45.1 
38. 7 
28.7 
22.5 
13.6 
8.5 
2.9 
0.9 
0.1 
0.0 



56 

52 - 
48 - 
44 - 
40 - 
36 - 
32 - 
28 - 
24 - 
20 - 
16 - 
12 - 
8 - 
4 - 
0 - 

- 4 - 

- 8 - 
-12 - 



55 
51 
47 
43 
39 
35 
31 
27 
23 
19 
15 
11 
7 
3 

- 1 

- 5 

- 9 



Standard 
Score 
Y 



Percentile 

Rank of 
Lower Limit 
of Interval 



15863 



91 


99,4 


93 


98.8 


35 


98.6 


69 


98.2 


121 


97.4 


244 


95.9 


400 


93.4 


957 


87.3 


2508 


71.5 


3698 


48.2 


35 75 


25.7 


2352 


10.8 


935 


4.9 


366 


2.6 


287 


0.8 


106 


0.2 


25 


0.01 


1 


0.00 



ERIC 



M 



12.25 



= 7.43 



M = 



o = 



Md = 12.37 
X 



(25 items) 



Conversion Data 



M = 19o93 



No conversion. 



= 8.43 



Conversion Data 
No conversion. 



M = 



0 = 



19.80 



X 

(56 items) 
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TEST ANALYSIS REPORT FORM 

Test National Longitudinal Study Subject 

Taken by Grade 12 students 



Form UEE 



Date Spring 1972 



Project 825 job 



50 



Mosaic Comparisons, Part 2 



Mosaic Comparisons, Part 3 



Raw 

Score 
X 



Standard 
Score 
Y 



Percentile 

Rank of 
Lower Limit 
of Interval 



Raw 
Score 
X 



Standard 
Score 
Y 



Percentile 

Rank of 
Lower Limit 
of Interval 



32 
30 
28 
26 
24 
22 
20 
18 
16 
14 
12 
10 
8 
6 
4 
2 
0 

- 2 

- 4 

- 6 

- 8 



33 
31 
29 
27 
25 
23 
21 
19 
17 
15 
13 
11 
9 
7 
5 
3 
1 

- 1 

- 3 

- 5 

7 



140 
53 
93 
159 
341 
632 
980 
1801 
2849 
2771 
2052 
1515 
945 
547 
264 
172 
368 
109 
63 
6 
3 

15863 



99.1 
98.8 
98.2 
97.2 
95.0 
91.1 
84.9 

73.5 , 

55.6 ■ 
38.1 
25.2 
15.6 

9.7 

6.2 

4.5 

3.5 

1.1 

0.5 

0.1 

0.02 

0.00 



26 - 
24 - 
22 - 
20 - 
18 - 
16 - 

12-'-'- 

10 - 

8 - 

6 - 

4 - 

2 - 

0 - 

- 2 - 

- 4 - 

- 6 - 

- 8 - 



27 
25 
23 
21 
19 
17 
15 
13 
11 
9 
7 
5 
3 
1 

- 1 

- 3 

- 5 

- 7 



120 
52 
111 
254 
6 74 
1018 
1975 
2495 
2664 
2919 
1610 
• 954 
402 
465 
118 
29 
2 
1 

15863 



99.2 

98.9 '■ 

98.2 

96.6 

92.4 

85.9 

73.5 

57.8 

41.0 

22.6 

12.4 
6.4 
3.9 
0.9 
0.2 
0.02 
0.01 
0.00 



ERIC 



M = 14.50 



= 5. 76 



M 



Md„ = 14.91 



(■\-\ Items') 



Conversion Data 
No conversion. 



M = 10.59 



Conversion Data 



= 4.84 



M 



lOV 



= 10,41 
X 

(2^teTns) 



No conversion 
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TEST ANALYSIS REPORT FORM 

Test Natio nal Longitudinal Study Subject 

Taken by Grade 12 students 



Form UEE 



Bate Spring 1972 



Project job _50_ 



ERIC 



Mosaic Comparisons, Tota 


.1 




Raw 
Score 
X 


Standard 
Score 
Y 


f 


r ercenu xxc 
rtariA ox 
->ower Limit 
01 xnuei^cii 


Rav7 
Score 
X 


Standard 
Score 
Y 


f 


r-i^rcenuixe 
Ranlc 01 
-lOwer Limit 
of Interval 


114 - 116 




43 


99.7 












108 - 114 




50 


99.4 












102 - 108 




14 


99.3 












96 - 102 




27 


99.2 












90 - 96 




36 


98.9 












84 - 90 




79 


98.4 












78 - 84 




145 ■ 


97.5 












72 - 78 




281 


95.7 












66 - 72 




. . 593 


92.0' 












60 - 66 




1112 


85.0 


- 










54 - 60 




1842 


73.4 












48 - 54 




2579 


57.1 












42 - 48 




2824 














36 - 42 




2361 


24.4 












30 - 3^ 




1658 


14.0 












24 - 30 




956 


8.0 












18 - 24 




528 


4.6 












12 - 18 




252 


3.0 












6 - 12 




152 


2.1 












0 - 6 




206 


0.8 












— b ~ - U 




93 


0.2 












- 12 - - 6 




24 


0.1 
















7 


0.01 












- 24 18 




1 


0.00 
















15863 














M = 45.01 


Conversion Data 


M 




Conversion Data 


X 






X 






0 = 16.43 


No conversion. 




0 






X 






X 






M = 






M 






y 






y 






0 = 






0 






y 






y 






Md ^ 45.10 






^d 






X 




X 






(116 items) 













Test National Longitudinal Study 

Description of Sample: 

Spaced sample 



Subject 



Form UEE 



N - 1,95.; 



Scoring Formulae and Reliability Coefficients for Sections 



Section of Test 



1 Vocabulary 



2 Picture-Number: 



Total 



3 Reading 



Scoring 
Formula 



R-W/4 



R-.lllW 



R-.lllW 



R-.lllW 



R-W/4 



Relia-. 
bility' 



.784 



,726** 



.845** 



,797 



SE 



me as 



1.96 



2.21 



2.14 



3.13 



2.32 



Section of Test 



4 Letter Groups 



5 Mathematics 



6 Mosaic Compar. : 



Scoring 
Formula 



R-W/4 



R-.333W 



R-W/2 



R-.333W 



R-W/4 



Relia-^ 
bility 



.861 



.866 



SE 



me as , 



2.23 



2.73 



*A:Iaptation of Kude r-Richardson formula (20). * *See text. 

Intercorrelati ons of Sections 



Section 


1 


2-1 


2-2 


2 


3 


4 


5 


6-1 


6-2 


6-3 


6 


1 1 Vocabulary 




.284 


.254 


.292 


.686 


.496 


.610 


.210 


.276 


.283 


.282 


2 Pic. -Num. : 1 


.284 




■' .726 


.933 


.351 


.448 


.413 


.305 


.354 


.338 


.372 


2 


.254 


.726 




.924 


.300 


.387 


.368 


.284 ■ 


.321 


.314 


.343 


Total 


.292 


.933 


.924 




.355 


.451 


.423 


.318 


.364 


.352 


.386 


3 Reading 


.686 


.351 


.300 


.355 




.595 


.667 


.274 


.342 


.342 


.354 


4 Letter Groups 


.496 


.448 


.387 


.451 


.595 




.674 


.387 


.488 


.460 


.494 


5 Mathematics 


o610 


.413 


.368 


.423 


.667 


.674 




.326 


.388 


.389 


.409 


6 Mosaic: 1 


.210 


.305' 


.284 


.318 


.274 


.387 


.326 




.683 


.579 


.900 


2 


.276 


.354 


.321 


.364 


.342 


.488 


.388 


.683 




.739 


.900 


3 


.283 


.338 


.314 


.352 


. 342 


.460 


.389 


.579 


.739 




.833 


Total 


.282 


.372 


.343 


.386 


. 354 


.494 


.409 


.900 


.900 


.833 










Speededness of Sections 




















1 


2-1 


2-2 


3 


4 


5 


6-1 


6-2 


6-3 






I'er cent com- 
pletlnf^ test.. 


82.3 






81.8 


57.6 


85.0 


1.3 


1.7 


2.0 






Per cent com- 
pleting 75 per 
cent of test . 


' 94.7- 






95.9 


94.7 


96.3 


2.5 


6.2 


6.1 




• 


|Number of items 
F reached by 80 
per cent of 
the candidates 


15 






20 


21 


25 


15 


12 


9 






Total number of 


15 


15 


15 


20 


25 


25 


56 


33 


•27 
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Test National Longitudinal Study 



-P- 

Subject 



Frequency Distributions of Original Deltas 
and Bi^erial Correlations, by Sections 



t 

O 
11 

< 

t 
•3 

CO 



ERIC 



Delta 


1 




0 
J 




A 












19.0 up . . 






















18.0-1S.9 






















17.0-17.9- 






















16.0-16.9 






1 




1 












15.0-15.9 


1 




















14.0-14.9 


4 




2 




1 




1 








13.0-13.9 


3 




5 








4 








12.0-1J2.9 


4 




4 




2 




9 








11.0-11.9 


~ 




1 




6 




2 








10.0-10.9 


1 




3 




5 




6 








9.0- 9.9 


1 




3 




3 




3 








8.0- 8.9 


1 




1 




2 












7.0- 7.9 










4 












6,0- 6.9 










1 












Total 


15 




20 




25 




25 










12.8 




12.1 




10.4 




11.7 










2.0 




2.0 




2.1 




1.4 






















































'^bis 




.90-. 99 






















.80-. 89 






















.70-. 79 


5 








9 




5 








.60-. 69 


6 




9 




9 




11 








.50-. 59 


2 




9 




6 




7 








.40-. 49 


1 




2 




1 




1 








.30-.39 


1 












1 








.20-. 29 






















.10-. 19 






















.00-. 09 






















Negative . 






















Total 


15 




20 




25 




25 








Not Comp. 
























.63 




.58 




.65 




.61 








f t 


.10 




.07 




.08 




.09 









APPENDIX D 
Item P-Values 
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